CN112269672B - File downloading exception handling method and device and electronic equipment - Google Patents
File downloading exception handling method and device and electronic equipment Download PDFInfo
- Publication number
- CN112269672B CN112269672B CN202011166383.1A CN202011166383A CN112269672B CN 112269672 B CN112269672 B CN 112269672B CN 202011166383 A CN202011166383 A CN 202011166383A CN 112269672 B CN112269672 B CN 112269672B
- Authority
- CN
- China
- Prior art keywords
- downloading
- file
- machine
- downloaded
- download
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0706—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment
- G06F11/0715—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment in a system implementing multitasking
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0751—Error or fault detection not based on redundancy
- G06F11/0754—Error or fault detection not based on redundancy by exceeding limits
- G06F11/0757—Error or fault detection not based on redundancy by exceeding limits by exceeding a time limit, i.e. time-out, e.g. watchdogs
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0766—Error or fault reporting or storing
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0766—Error or fault reporting or storing
- G06F11/0787—Storage of error reports, e.g. persistent data storage, storage using memory protection
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a file downloading exception handling method, a device and electronic equipment, wherein the method comprises the following steps: storing a downloading task corresponding to the ID of the file to be downloaded into a downloading queue; calling a distributed task scheduling system to read a plurality of downloading tasks from the downloading queue, and distributing the plurality of downloading tasks to a plurality of machines for simultaneous execution; processing the file downloading and the read abnormal information in parallel by adopting a plurality of threads, and storing the abnormal information of the downloading task executed by the machines into a database; and acquiring thread heartbeat for executing the abnormality according to the abnormality information, and sending a downloading abnormality report. The invention distributes a plurality of downloading tasks to a plurality of machines for simultaneous execution through the distributed task scheduling system, adopts a plurality of threads to capture various abnormal information of file reading and downloading in real time, and processes the abnormal information of file downloading and reading in parallel, thereby realizing timely compensation for the abnormality in file downloading or reading and reducing data loss.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method and apparatus for processing file downloading abnormality, an electronic device, and a computer readable medium.
Background
During the operation of the system (such as an operating system, a software system and the like), the operation environment of the system and the system itself may influence the operation of the system, such as hardware resources, hardware working conditions, network communication, errors of the system itself, operation defects of the system, environmental temperature and the like, may cause abnormal operation of the system, thereby causing downtime.
For a list pushed from an external system (such as some apps), if a server suddenly goes down during the process of reading or downloading a file, data is lost, so that abnormal data downloading occurs.
In the prior art, abnormal information of data downloading needs to be acquired from a database at regular time, and the reasons of abnormal downloading cannot be positioned timely and accurately due to the timeliness of the abnormal information, so that lost data cannot be compensated, and the integrity of the downloaded data is affected.
Disclosure of Invention
The invention aims to solve the technical problem that the abnormal download data is lost and is difficult to compensate because the abnormal download data cannot be positioned in time in the file downloading process.
In order to solve the above technical problems, a first aspect of the present invention provides a method for handling file downloading abnormality, the method comprising:
receiving a file ID to be downloaded sent by an external system;
storing the downloading task corresponding to the file ID to be downloaded into a downloading queue;
calling a distributed task scheduling system to read a plurality of downloading tasks from the downloading queue, and distributing the plurality of downloading tasks to a plurality of machines for simultaneous execution;
processing the file downloading and the read abnormal information in parallel by adopting a plurality of threads, and storing the abnormal information of the downloading task executed by the machines into a database;
and acquiring thread heartbeat for executing the abnormality according to the abnormality information, and sending a downloading abnormality report.
According to a preferred embodiment of the present invention, the distributing the plurality of download tasks to a plurality of machines for simultaneous execution comprises:
the plurality of download tasks are distributed to the plurality of machines in an average manner;
assigning a remaining number of download tasks to the specified machine;
and controlling the multiple machines to simultaneously execute the downloading task.
According to a preferred embodiment of the present invention, the processing, in parallel, of the exception information of file downloading and reading using a plurality of threads includes:
if the downloading of a certain machine file fails, controlling the machine to download the file with the failed downloading again;
if the downloading of the file of a certain machine is successful, but the file reading fails, the machine is controlled to read the downloaded file again from the next line of the current reading line number.
According to a preferred embodiment of the present invention, the storing the abnormality information of the plurality of machines for performing the download task in the database includes:
if the downloading of a certain machine file fails, storing the machine ID, the downloaded file ID and the downloading time into a database;
if a certain machine file is successfully downloaded, but the file is not read, storing the machine ID, the downloaded file ID, the current reading line number and the downloading time into a database;
if a certain machine file is successfully downloaded and the file is successfully read, the downloaded file ID in the machine ID in the database is cleared.
According to a preferred embodiment of the present invention, the acquiring the thread heartbeat for executing the exception according to the exception information, and sending the download exception report includes:
acquiring the downloading time in the current abnormal information from the database;
judging whether the heartbeat of a thread executing a corresponding downloading task is abnormal or not according to the current time and the downloading time;
and if the thread heartbeat is abnormal, sending an abnormal report.
According to a preferred embodiment of the present invention, the distributed task scheduling system is any one of xxl-job, cronsun, elastic-job, saturn, TBSchedule.
According to a preferred embodiment of the invention, the database is any one of Redis, oracle, sybase, informix.
In order to solve the above technical problem, a second aspect of the present invention provides a file downloading exception handling apparatus, the apparatus including:
the receiving module is used for receiving the file ID to be downloaded sent by the external system;
the first storage module is used for storing the downloading task corresponding to the file ID to be downloaded into a downloading queue;
the distribution module is used for calling the distributed task scheduling system to read a plurality of download tasks from the download queue and distributing the plurality of download tasks to a plurality of machines for simultaneous execution;
the second storage module is used for processing the file downloading and the read abnormal information in parallel by adopting a plurality of threads and storing the abnormal information of the downloading task executed by the plurality of machines into a database;
and the sending module is used for acquiring the thread heartbeat for executing the abnormality according to the abnormality information and sending a downloading abnormality report.
According to a preferred embodiment of the invention, the dispensing module comprises:
a first allocation module for equally allocating the plurality of download tasks to the plurality of machines;
a second allocation module for allocating a remaining number of download tasks to the specified machine;
and the control module is used for controlling the plurality of machines to simultaneously execute the downloading task.
According to a preferred embodiment of the present invention, the second storage module includes: the abnormal processing module is used for controlling a certain machine to download the file with failed downloading if the downloading of the file of the machine fails; if the downloading of the file of a certain machine is successful, but the file reading fails, the machine is controlled to read the downloaded file again from the next line of the current reading line number.
According to a preferred embodiment of the present invention, the second storage module further includes: the sub-storage module is used for storing the machine ID, the downloaded file ID and the downloading time into a database if the downloading of a certain machine file fails; if a certain machine file is successfully downloaded, but the file is not read, storing the machine ID, the downloaded file ID, the current reading line number and the downloading time into a database; if a certain machine file is successfully downloaded and the file is successfully read, the downloaded file ID in the machine ID in the database is cleared.
According to a preferred embodiment of the present invention, the transmitting module includes:
the acquisition module is used for acquiring the downloading time in the current abnormal information from the database;
the judging module is used for judging whether the heartbeat of the thread executing the corresponding downloading task is abnormal or not according to the current time and the downloading time;
and the sub-sending module is used for sending an exception report if the thread heartbeat is abnormal.
According to a preferred embodiment of the present invention, the distributed task scheduling system is any one of xxl-job, cronsun, elastic-job, saturn, TBSchedule.
According to a preferred embodiment of the invention, the database is any one of Redis, oracle, sybase, informix.
To solve the above technical problem, a third aspect of the present invention provides an electronic device, including:
a processor; and
a memory storing computer executable instructions that, when executed, cause the processor to perform the method described above.
To solve the above technical problem, a fourth aspect of the present invention provides a computer-readable storage medium storing one or more programs, which when executed by a processor, implement the above method.
The distributed task scheduling system is combined with the multithreading downloading, a plurality of downloading tasks are read from the downloading queue through the distributed task scheduling system, the plurality of downloading tasks are distributed to a plurality of machines to be executed simultaneously, various abnormal information of file reading and downloading is captured in real time by adopting a plurality of threads, the abnormal information of file downloading and reading is processed in parallel, timely compensation of abnormality in file downloading or reading is realized, and data loss caused by data downloading abnormality is effectively reduced. Simultaneously storing the abnormal information of the plurality of thread downloading tasks to a database; and acquiring thread heartbeat for executing the abnormality according to the abnormality information, sending a downloading abnormality report, and recording the abnormality thread information, so that the subsequent repair is convenient.
Drawings
In order to make the technical problems solved by the present invention, the technical means adopted and the technical effects achieved more clear, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be noted, however, that the drawings described below are merely illustrative of exemplary embodiments of the present invention and that other embodiments of the drawings may be derived from these drawings by those skilled in the art without undue effort.
FIG. 1 is a flow chart of a method for handling file downloading exceptions according to the present invention;
FIG. 2 is a schematic diagram of file download exception handling in accordance with the present invention;
FIG. 3 is a schematic diagram of a file downloading exception handling apparatus according to the present invention;
FIG. 4 is a block diagram of an exemplary embodiment of an electronic device in accordance with the present invention;
FIG. 5 is a schematic diagram of one embodiment of a computer readable medium of the present invention.
Detailed Description
Exemplary embodiments of the present invention will now be described more fully hereinafter with reference to the accompanying drawings, in which exemplary embodiments are shown, although the exemplary embodiments may be practiced in various specific ways. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the invention to those skilled in the art.
The structures, capabilities, effects, or other features described in a particular embodiment may be incorporated in one or more other embodiments in any suitable manner without departing from the spirit of the present invention.
In describing particular embodiments, specific details of construction, performance, effects, or other features are set forth in order to provide a thorough understanding of the embodiments by those skilled in the art. It is not excluded, however, that one skilled in the art may implement the present invention in a particular situation in a solution that does not include the structures, properties, effects, or other characteristics described above.
The flow diagrams in the figures are merely exemplary flow illustrations and do not represent that all of the elements, operations, and steps in the flow diagrams must be included in the aspects of the present invention, nor that the steps must be performed in the order shown in the figures. For example, some operations/steps in the flowcharts may be decomposed, some operations/steps may be combined or partially combined, etc., and the order of execution shown in the flowcharts may be changed according to actual situations without departing from the gist of the present invention.
The block diagrams in the figures generally represent functional entities and do not necessarily correspond to physically separate entities. That is, the functional entities may be implemented in software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.
The same reference numerals in the drawings denote the same or similar elements, components or portions, and thus repeated descriptions of the same or similar elements, components or portions may be omitted hereinafter. It will be further understood that, although the terms first, second, third, etc. may be used herein to describe various devices, elements, components or portions, these devices, elements, components or portions should not be limited by these terms. That is, these phrases are merely intended to distinguish one from the other. For example, a first device may also be referred to as a second device without departing from the spirit of the invention. Furthermore, the term "and/or," "and/or" is meant to include all combinations of any one or more of the items listed.
Referring to fig. 1, fig. 1 is a flowchart of a method for handling file downloading abnormality according to the present invention. As shown in fig. 1, the method includes:
s1, receiving a file ID to be downloaded, which is sent by an external system;
wherein the external system may be a service system such as: financial lending business system, insurance business system, collection system, logistics business system and the like; or some APP application. In this embodiment, the external system may call the list pushing service of the list system through the http interface, and the external system may provide a plurality of file IDs to be downloaded.
S2, storing the downloading task corresponding to the file ID to be downloaded into a downloading queue;
wherein, a file ID to be downloaded corresponds to a unique download task. The downloading task comprises the following steps: file ID to be downloaded, downloading mode, serial number of request, number of file ID to be downloaded, etc.
In order to prevent repeated submitting of the downloading task, the step is to perform idempotent processing on the downloading task according to the serial number of the request. The download tasks processed by idempotent are stored in a download queue, which may be a redis queue in particular.
S3, calling a distributed task scheduling system to read a plurality of downloading tasks from the downloading queue, and distributing the plurality of downloading tasks to a plurality of machines for simultaneous execution;
wherein the distributed task scheduling system can be any one of xxl-job, cronsun, elastic-job, saturn, TBSchedule. The invention prefers xxl-job, then this step obtains at least one download task from the redis queue that needs to be downloaded through the slice broadcast policy of xxljob.
As in fig. 2, for a plurality of download tasks, xxljob distributes the plurality of download tasks equally to a plurality of machines; assigning a remaining number of download tasks to the specified machine; and controlling the multiple machines to simultaneously execute the downloading task. Wherein the number of download tasks equally allocated = download task summary/machine number. Taking the example of 56 download tasks being assigned to 5 machines, 11 download tasks are equally assigned to the 5 machines and the remaining 1 download task is assigned to a designated machine, such as the first machine. After the download task is distributed, the 5 machines are controlled to execute the download task simultaneously.
S4, processing the file downloading and the read abnormal information in parallel by adopting a plurality of threads, and storing the abnormal information of the downloading task executed by the machines into a database;
in this step, each thread downloads and reads the file from the file source server according to the file ID to be downloaded corresponding to the download task, and processes the file downloading and the read exception information in parallel.
Wherein the anomaly information includes: failure in file download, successful file download but failure in file reading. If the downloading of a certain machine file fails, controlling the machine to download the file with the failed downloading again; if the downloading of the file of a certain machine is successful, but the file reading fails, the machine is controlled to read the downloaded file again from the next line of the current reading line number. By the method, the reasons for abnormal file downloading can be found in time, and corresponding measures are taken in time to make up, so that data loss caused by abnormal data downloading is reduced.
Further, if the downloading of a certain machine file fails, storing the machine ID, the downloaded file ID and the downloading time into a database; if a certain machine file is successfully downloaded, but the file is not read, storing the machine ID, the downloaded file ID, the current reading line number and the downloading time into a database. In the invention, the current reading line number is recorded and stored in a batch in ten thousands of ways when the reading fails. If a certain machine file is successfully downloaded and the file is successfully read, the downloaded file ID in the machine ID in the database is cleared. As shown in fig. 2, the download tasks are distributed to the machine 1, the machine 2 and the machine 3, if the machine 1 fails to download the file, the machine 1 is controlled to download the file again, and the ID of the machine 1, the downloaded file ID and the download time are stored in a database; if the machine 2 file download is successful and the file read is successful, the downloaded file ID in the ID of machine 2 in the database is cleared. If the machine 3 downloads the file successfully, but the file is not read, the machine 3 is controlled to read the downloaded file again from the next line of the current reading line number, and simultaneously the ID of the machine 3, the downloaded file ID, the current reading line number and the downloading time are stored in the database.
Wherein the database may be any one of Redis, oracle, sybase, informix. The database in the present invention is preferably a Redis database. The invention stores and downloads the abnormal information through the Redis database, and can judge whether the thread heartbeat processing is abnormal or not through the abnormal information.
S5, acquiring thread heartbeat for executing the abnormality according to the abnormality information, and sending a downloading abnormality report.
Specifically, the method comprises the following steps:
s501, acquiring the downloading time in the current abnormal information from the database;
in the step, the downloading time in the current abnormal information is obtained from the database in real time. The downloading time refers to a time point when an abnormality occurs in a downloading task.
S502, judging whether the heartbeat of a thread executing a corresponding downloading task is abnormal or not according to the current time and the downloading time;
specifically, if the time interval between the current time and the download time is greater than the preset time interval, determining that the thread heartbeat for executing the download task corresponding to the download time is abnormal. Wherein the preset time interval is preferably 30 minutes.
S503, if the thread heartbeat is abnormal, sending an abnormal report.
Specifically, a report of thread heartbeat anomaly may be sent to the download manager; the report of the thread heartbeat abnormality can also be sent in a log downloading mode.
Further, the method may further include:
and downloading and reading the file and the list information, and pushing the file and the list information to a downstream system.
FIG. 3 is a schematic diagram of a file downloading exception handling apparatus according to the present invention, as shown in FIG. 3, the apparatus includes:
a receiving module 31, configured to receive a file ID to be downloaded sent by an external system;
a first storage module 32, configured to store a download task corresponding to the file ID to be downloaded into a download queue;
the allocation module 33 is configured to invoke the distributed task scheduling system to read a plurality of download tasks from the download queue, and allocate the plurality of download tasks to a plurality of machines for simultaneous execution;
the second storage module 34 is configured to process the file downloading and the read exception information in parallel by using multiple threads, and store the exception information of the downloading task executed by the multiple machines into a database;
and a sending module 35, configured to obtain, according to the anomaly information, a thread heartbeat for performing anomaly, and send a download anomaly report.
In one embodiment, the distribution module 33 includes:
a first allocation module 331, configured to equally allocate the plurality of download tasks to the plurality of machines;
a second allocation module 332 for allocating a remaining number of download tasks to a specified machine;
a control module 333, configured to control the multiple machines to perform the downloading task simultaneously.
The second storage module 34 includes:
the exception handling module 341 is configured to, if a certain machine fails to download a file, control the machine to download the file that fails to download again; if the downloading of the file of a certain machine is successful, but the file reading fails, the machine is controlled to read the downloaded file again from the next line of the current reading line number.
A sub-storage module 342, configured to store a machine ID, a downloaded file ID, and a download time into a database if a download of a machine file fails; if a certain machine file is successfully downloaded, but the file is not read, storing the machine ID, the downloaded file ID, the current reading line number and the downloading time into a database; if a certain machine file is successfully downloaded and the file is successfully read, the downloaded file ID in the machine ID in the database is cleared.
The transmitting module 35 includes:
an obtaining module 351, configured to obtain a download time in the current abnormal information from the database;
the judging module 352 is configured to judge whether the heartbeat of the thread executing the corresponding download task is abnormal according to the current time and the download time;
the sub-sending module 353 is configured to send an exception report if the thread heartbeat is abnormal.
In the invention, the distributed task scheduling system is any one of xxl-job, cronsun, elastic-job, saturn, TBSchedule. The database is any one of Redis, oracle, sybase, informix.
It will be appreciated by those skilled in the art that the modules in the embodiments of the apparatus described above may be distributed in an apparatus as described, or may be distributed in one or more apparatuses different from the embodiments described above with corresponding changes. The modules of the above embodiments may be combined into one module, or may be further split into a plurality of sub-modules.
The following describes an embodiment of an electronic device of the present invention, which may be regarded as a physical form of implementation for the above-described embodiment of the method and apparatus of the present invention. Details described in relation to the embodiments of the electronic device of the present invention should be considered as additions to the embodiments of the method or apparatus described above; for details not disclosed in the embodiments of the electronic device of the present invention, reference may be made to the above-described method or apparatus embodiments.
Fig. 4 is a block diagram of an exemplary embodiment of an electronic device according to the present invention. The electronic device shown in fig. 4 is only an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.
As shown in fig. 4, the electronic device 400 of the exemplary embodiment is in the form of a general-purpose data processing device. The components of electronic device 400 may include, but are not limited to: at least one processing unit 410, at least one memory unit 420, a bus 430 connecting the different electronic device components (including memory unit 420 and processing unit 410), a display unit 440, and the like.
The storage unit 420 stores a computer readable program, which may be a source program or code of a read only program. The program may be executed by the processing unit 410 such that the processing unit 410 performs the steps of various embodiments of the present invention. For example, the processing unit 410 may perform the steps shown in fig. 1.
The memory unit 420 may include readable media in the form of volatile memory units, such as Random Access Memory (RAM) 4201 and/or cache memory 4202, and may further include Read Only Memory (ROM) 4203. The storage unit 420 may also include a program/utility 4204 having a set (at least one) of program modules 4205, such program modules 4205 including, but not limited to: an operating electronic device, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.
The electronic device 400 may also communicate with one or more external devices 300 (e.g., keyboard, display, network device, bluetooth device, etc.), such that a user can interact with the electronic device 400 via the external devices 300, and/or such that the electronic device 400 can communicate with one or more other data processing devices (e.g., routers, modems, etc.). Such communication may occur through an input/output (I/O) interface 450, and may also occur through a network adapter 460 to one or more networks, such as a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the internet. The network adapter 460 may communicate with other modules of the electronic device 400 via the bus 430. It should be appreciated that although not shown in fig. 4, other hardware and/or software modules may be used in electronic device 400, including, but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID electronics, tape drives, data backup storage electronics, and the like.
FIG. 5 is a schematic diagram of one embodiment of a computer readable medium of the present invention. As shown in fig. 5, the computer program may be stored on one or more computer readable media. The computer readable medium may be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electronic device, apparatus, or means of electronic, magnetic, optical, electromagnetic, infrared, or semiconductor, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. The computer program, when executed by one or more data processing devices, enables the computer readable medium to carry out the above-described method of the present invention, namely: receiving a file ID to be downloaded sent by an external system; storing the downloading task corresponding to the file ID to be downloaded into a downloading queue; calling a distributed task scheduling system to read a plurality of downloading tasks from the downloading queue, and distributing the plurality of downloading tasks to a plurality of machines for simultaneous execution; processing the file downloading and the read abnormal information in parallel by adopting a plurality of threads, and storing the abnormal information of the downloading task executed by the machines into a database; and acquiring thread heartbeat for executing the abnormality according to the abnormality information, and sending a downloading abnormality report.
From the above description of embodiments, those skilled in the art will readily appreciate that the exemplary embodiments described herein may be implemented in software, or may be implemented in software in combination with necessary hardware. Thus, the technical solution according to the embodiments of the present invention may be embodied in the form of a software product, which may be stored in a computer readable storage medium (may be a CD-ROM, a usb disk, a mobile hard disk, etc.) or on a network, comprising several instructions to cause a data processing device (may be a personal computer, a server, or a network device, etc.) to perform the above-described method according to the present invention.
The computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, with readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A readable storage medium may also be any readable medium that can communicate, propagate, or transport a program for use by or in connection with an instruction execution electronic device, apparatus, or device. Program code embodied on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).
In summary, the present invention may be implemented in a method, apparatus, electronic device, or computer readable medium that executes a computer program. Some or all of the functions of the present invention may be implemented in practice using a general purpose data processing device such as a microprocessor or Digital Signal Processor (DSP).
The above-described specific embodiments further describe the objects, technical solutions and advantageous effects of the present invention in detail, and it should be understood that the present invention is not inherently related to any particular computer, virtual device or electronic apparatus, and various general-purpose devices may also implement the present invention. The foregoing description of the embodiments of the invention is not intended to be limiting, but rather is intended to cover all modifications, equivalents, alternatives, and improvements that fall within the spirit and scope of the invention.
Claims (14)
1. A method for handling file download exceptions, the method comprising:
receiving a file ID to be downloaded sent by an external system;
storing the downloading task corresponding to the file ID to be downloaded into a downloading queue;
calling a distributed task scheduling system to read a plurality of downloading tasks from the downloading queue, and distributing the plurality of downloading tasks to a plurality of machines for simultaneous execution;
processing the file downloading and the read abnormal information in parallel by adopting a plurality of threads, and storing the abnormal information of the downloading task executed by the machines into a database;
acquiring thread heartbeat for executing abnormality according to the abnormality information, and sending a downloading abnormality report;
wherein: the storing the abnormal information of the downloading task executed by the machines into a database comprises:
if the downloading of a certain machine file fails, storing the machine ID, the downloaded file ID and the downloading time into a database;
if a certain machine file is successfully downloaded, but the file is not read, storing the machine ID, the downloaded file ID, the current reading line number and the downloading time into a database;
if a certain machine file is successfully downloaded and the file is successfully read, the downloaded file ID in the machine ID in the database is cleared.
2. The method of claim 1, wherein the assigning the plurality of download tasks to a plurality of machines is performed simultaneously comprises:
the plurality of download tasks are distributed to the plurality of machines in an average manner;
assigning a remaining number of download tasks to the specified machine;
and controlling the multiple machines to simultaneously execute the downloading task.
3. The method of claim 1, wherein processing the file download and read exception information in parallel using a plurality of threads comprises:
if the downloading of a certain machine file fails, controlling the machine to download the file with the failed downloading again;
if the downloading of the file of a certain machine is successful, but the file reading fails, the machine is controlled to read the downloaded file again from the next line of the current reading line number.
4. The method of claim 3, wherein the acquiring a thread heartbeat for performing an exception based on the exception information and sending a download exception report comprises:
acquiring the downloading time in the current abnormal information from the database;
judging whether the heartbeat of a thread executing a corresponding downloading task is abnormal or not according to the current time and the downloading time;
and if the thread heartbeat is abnormal, sending an abnormal report.
5. The method of claim 1, wherein the distributed task scheduling system is any one of xxl-job, cronsun, elastic-job, saturn, TBSchedule.
6. The method of claim 1, wherein the database is any one of Redis, oracle, sybase, informix.
7. A file download exception handling apparatus, said apparatus comprising:
the receiving module is used for receiving the file ID to be downloaded sent by the external system;
the first storage module is used for storing the downloading task corresponding to the file ID to be downloaded into a downloading queue;
the distribution module is used for calling the distributed task scheduling system to read a plurality of download tasks from the download queue and distributing the plurality of download tasks to a plurality of machines for simultaneous execution;
the second storage module is used for processing the file downloading and the read abnormal information in parallel by adopting a plurality of threads and storing the abnormal information of the downloading task executed by the plurality of machines into a database;
the sending module is used for acquiring thread heartbeat for executing the abnormality according to the abnormality information and sending a downloading abnormality report;
wherein: the second storage module includes: the sub-storage module is used for storing the machine ID, the downloaded file ID and the downloading time into a database if the downloading of a certain machine file fails; if a certain machine file is successfully downloaded, but the file is not read, storing the machine ID, the downloaded file ID, the current reading line number and the downloading time into a database; if a certain machine file is successfully downloaded and the file is successfully read, the downloaded file ID in the machine ID in the database is cleared.
8. The apparatus of claim 7, wherein the allocation module comprises:
a first allocation module for equally allocating the plurality of download tasks to the plurality of machines;
a second allocation module for allocating a remaining number of download tasks to the specified machine;
and the control module is used for controlling the plurality of machines to simultaneously execute the downloading task.
9. The apparatus of claim 7, wherein the second storage module comprises: the abnormal processing module is used for controlling a certain machine to download the file with failed downloading if the downloading of the file of the machine fails; if the downloading of the file of a certain machine is successful, but the file reading fails, the machine is controlled to read the downloaded file again from the next line of the current reading line number.
10. The apparatus of claim 9, wherein the transmitting module comprises:
the acquisition module is used for acquiring the downloading time in the current abnormal information from the database;
the judging module is used for judging whether the heartbeat of the thread executing the corresponding downloading task is abnormal or not according to the current time and the downloading time;
and the sub-sending module is used for sending an exception report if the thread heartbeat is abnormal.
11. The apparatus of claim 7, wherein the distributed task scheduling system is any one of xxl-job, cronsun, elastic-job, saturn, TBSchedule.
12. The apparatus of claim 7, wherein the database is any one of Redis, oracle, sybase, informix.
13. An electronic device, comprising:
a processor; and
a memory storing computer-executable instructions that, when executed, cause the processor to perform the method of any of claims 1-6.
14. A computer readable storage medium storing one or more programs, which when executed by a processor, implement the method of any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011166383.1A CN112269672B (en) | 2020-10-27 | 2020-10-27 | File downloading exception handling method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011166383.1A CN112269672B (en) | 2020-10-27 | 2020-10-27 | File downloading exception handling method and device and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112269672A CN112269672A (en) | 2021-01-26 |
| CN112269672B true CN112269672B (en) | 2023-05-02 |
Family
ID=74342395
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011166383.1A Active CN112269672B (en) | 2020-10-27 | 2020-10-27 | File downloading exception handling method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112269672B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114595082B (en) * | 2022-03-07 | 2025-07-15 | 天津爱笔智能科技有限公司 | A method and device for processing data sets |
| CN116166624A (en) * | 2023-01-03 | 2023-05-26 | 上海中通吉网络技术有限公司 | File import method, device and equipment |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106302709A (en) * | 2016-08-12 | 2017-01-04 | 杭州东信北邮信息技术有限公司 | A kind of method and system of network file management |
| CN107347092A (en) * | 2017-06-30 | 2017-11-14 | 环球智达科技（北京）有限公司 | The method downloaded for multithreading |
| CN110572422A (en) * | 2018-06-06 | 2019-12-13 | 北京京东尚科信息技术有限公司 | Data downloading method and device |
| CN111741112A (en) * | 2020-06-22 | 2020-10-02 | 中国平安财产保险股份有限公司 | File downloading method, device, equipment and storage medium based on artificial intelligence |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6769019B2 (en) * | 1997-12-10 | 2004-07-27 | Xavier Ferguson | Method of background downloading of information from a computer network |
-
2020
- 2020-10-27 CN CN202011166383.1A patent/CN112269672B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106302709A (en) * | 2016-08-12 | 2017-01-04 | 杭州东信北邮信息技术有限公司 | A kind of method and system of network file management |
| CN107347092A (en) * | 2017-06-30 | 2017-11-14 | 环球智达科技（北京）有限公司 | The method downloaded for multithreading |
| CN110572422A (en) * | 2018-06-06 | 2019-12-13 | 北京京东尚科信息技术有限公司 | Data downloading method and device |
| CN111741112A (en) * | 2020-06-22 | 2020-10-02 | 中国平安财产保险股份有限公司 | File downloading method, device, equipment and storage medium based on artificial intelligence |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112269672A (en) | 2021-01-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8225308B2 (en) | Managing software lifecycle | |
| CN114610499B (en) | Task scheduling method, device, computer readable storage medium and electronic device | |
| US9389916B1 (en) | Job scheduling management | |
| US8301935B2 (en) | Distributed batch runner | |
| CN110851324B (en) | Log-based routing inspection processing method and device, electronic equipment and storage medium | |
| CN112269672B (en) | File downloading exception handling method and device and electronic equipment | |
| CN111831503B (en) | Monitoring method based on monitoring agent and monitoring agent device | |
| CN111324474B (en) | Celery task management method, system, equipment and storage medium | |
| CN110807058A (en) | Method and system for exporting data | |
| CN112925721A (en) | Distributed system testing method and device | |
| CN113360368A (en) | Method and device for testing software performance | |
| CN113407429A (en) | Task processing method and device | |
| CN110896362A (en) | Fault detection method and device | |
| US20140195662A1 (en) | Management of mobile applications in communication networks | |
| CN113763023B (en) | User identification method, device, electronic equipment and storage medium | |
| CN111159010A (en) | Defect collecting method, defect reporting method and related device | |
| CN113138807B (en) | Method and device for executing multi-node service task and electronic equipment | |
| CN113986097A (en) | Task scheduling method and device and electronic equipment | |
| US11115275B2 (en) | Systems and methods for automated technology resource decommissioning | |
| CN115080434A (en) | Case execution method, apparatus, equipment and medium | |
| CN113961388A (en) | Application crash preprocessing method and crash handling method | |
| CN113807817A (en) | Operation and maintenance management method and system, storage medium and electronic equipment | |
| CN112559001A (en) | Method and device for updating application | |
| CN112181701B (en) | Method and device for locating abnormal service request | |
| US9436523B1 (en) | Holistic non-invasive evaluation of an asynchronous distributed software process |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP01 | Change in the name or title of a patent holder | ||
| CP01 | Change in the name or title of a patent holder |
Address after: 201500 room a3-5588, 58 Fumin Branch Road, Hengsha Township, Chongming District, Shanghai (Shanghai Hengtai Economic Development Zone) Patentee after: Qifu Shuke (Shanghai) Technology Co.,Ltd. Address before: 201500 room a3-5588, 58 Fumin Branch Road, Hengsha Township, Chongming District, Shanghai (Shanghai Hengtai Economic Development Zone) Patentee before: Shanghai Qifu Information Technology Co.,Ltd. |