CN106095554A - The method and device of batch data process is carried out in the most online stage - Google Patents
The method and device of batch data process is carried out in the most online stage Download PDFInfo
- Publication number
- CN106095554A CN106095554A CN201610439380.8A CN201610439380A CN106095554A CN 106095554 A CN106095554 A CN 106095554A CN 201610439380 A CN201610439380 A CN 201610439380A CN 106095554 A CN106095554 A CN 106095554A
- Authority
- CN
- China
- Prior art keywords
- request
- processes
- batch
- batch service
- queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/02—Banking, e.g. interest calculation or account maintenance
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Accounting & Taxation (AREA)
- Finance (AREA)
- Software Systems (AREA)
- General Physics & Mathematics (AREA)
- Development Economics (AREA)
- Technology Law (AREA)
- General Business, Economics & Management (AREA)
- Strategic Management (AREA)
- Marketing (AREA)
- Economics (AREA)
- General Engineering & Computer Science (AREA)
- Financial Or Insurance-Related Operations Such As Payment And Settlement (AREA)
Abstract
The invention provides a kind of method and device carrying out batch data process in the most online stage, wherein, method includes: process request in the online business middle batch service that receives alternately；The process request of described batch service is divided into the queue that processing priority is different by the mark processing request according to described batch service；During online business processes, sequentially process described batch service on backstage according to the priority of described queue and process request.The present invention can allow operator just can view the result of batch data business at current trading day, improves the satisfaction of client.
Description
Technical field
The present invention relates to field of computer technology, particularly relate to a kind of big data processing method quasi real time, specifically
It it is exactly a kind of method and device carrying out batch data process in the most online stage.
Background technology
It is known that banking is divided into online transaction business, (online transaction business refers to reception and the place of service request
Reason is all real-time, in the very first time, result will be returned to requesting party, i.e. requirement of real-time after once having processed
High business, such as, withdrawal transaction, money transfer transactions etc.) and batch processing business (batch processing business refers to the business of user
Request is set about processing after being received the most immediately, but after running up to a certain degree, together submission processes, then computer
According to rule ordering, these service request are processed one by one, after all request is disposed again by result in the lump
Output, the business that i.e. requirement of real-time is relatively low, such as, wage keeps accounts, report generation, account balance historical record etc.).In the daytime
Online phase process online transaction business, carries out batch processing business in the stage at night.Batch processing business typically requires and carries out
Calculating in a large number, the longest and consumption duration is not fixed, and general backstage of passing through processed in batches at night, and by result
It is supplied to operator in T+1 day (i.e. next day of current trading day)；And online transaction business processes and usually requires that banking
Business processing system can quickly respond and return in real time, allows the operator to obtain as early as possible result.
But under a little situations, operator need to obtain the process knot of batch processing business at current trading day (i.e. T day)
Really, owing to operator cannot directly initiate on foreground, at this moment have to allow the technical staff possessing system maintenance right rear
Platform is directly submitted to, poor operability.And, batch processing business can long-time occupying system resources, due to system resource
Can not get release for a long time, the process of online transaction business certainly will be had influence on, there is the on-line transaction production safety risk such as unsuccessfully, shadow
Ring the carrying out of bank's routine work.
Therefore, how meet online transaction business process that concurrency is big, response time short on the premise of, stably carry out
The big data of batch process, and just can return result in T day, are the long-term problem demanding prompt solutions of those skilled in the art.
Summary of the invention
In view of this, the technical problem to be solved in the present invention is to provide one to carry out batch data in the most online stage
The method and device processed, solving cannot be in the problem of the result obtaining batch service the same day of concluding the business in prior art.
In order to solve above-mentioned technical problem, the detailed description of the invention of the present invention provides a kind of and carries out criticizing in the most online stage
The method that amount data process, including: process request in the online business middle batch service that receives alternately；At described batch service
Described batch service is processed request and is divided into the queue that processing priority is different by the mark of reason request；Preferential according to described queue
Level sequentially processes described batch service during online business processes and processes request on backstage.
The detailed description of the invention of the present invention also provides for a kind of device carrying out batch data process in the most online stage, bag
Include: receive unit, for processing request in the online business middle batch service that receives alternately；Queue assignment unit, for according to institute
The process request of described batch service is divided into the queue that processing priority is different by the mark stating batch service process request；Process single
Unit, please for sequentially processing the process of described batch service during online business processes on backstage according to the priority of described queue
Ask.
According to the above-mentioned detailed description of the invention of the present invention, in the method that the most online stage carries out batch data process
And device at least has the advantages that the process request receiving batch service in online transaction (online business) is mutual,
On backstage, batch processing business datum is carried out data mart modeling, be not take up the time window of online transaction, and operator are at T
Day (current trading day) just can view the result of batch service；Priority according to batch data business is in background process
Batch data business, and utilize dynamic priority algorithm based on batch data business throughput and the time of process in background process
Lowest priority other batch data business, it is possible to the most online window of Appropriate application so that each batch data service request
Having the opportunity to be processed, operator just can view the result of batch data business at current trading day, improves and turns round and look at
The satisfaction of visitor.
It is to be understood that above-mentioned general description and detailed description below be merely illustrative and illustrative, it is not
The scope that the present invention to be advocated can be limited.
Accompanying drawing explanation
Following appended accompanying drawing is a part for the description of the present invention, and it depicts the example embodiment of the present invention, institute
Accompanying drawing is used for illustrating the principle of the present invention together with the description of description.
A kind of method carrying out batch data process in the most online stage that Fig. 1 provides for the specific embodiment of the invention
The flow chart of specific embodiment one；
A kind of method carrying out batch data process in the most online stage that Fig. 2 provides for the specific embodiment of the invention
The flow chart of specific embodiment two；
A kind of method carrying out batch data process in the most online stage that Fig. 3 provides for the specific embodiment of the invention
The flow chart of specific embodiment three；
A kind of method carrying out batch data process in the most online stage that Fig. 4 provides for the specific embodiment of the invention
The flow chart of specific embodiment four；
A kind of device carrying out batch data process in the most online stage that Fig. 5 provides for the specific embodiment of the invention
The schematic block diagram of specific embodiment one；
A kind of device carrying out batch data process in the most online stage that Fig. 6 provides for the specific embodiment of the invention
The schematic block diagram of specific embodiment two；
A kind of device carrying out batch data process in the most online stage that Fig. 7 provides for the specific embodiment of the invention
The schematic block diagram of specific embodiment three；
A kind of device carrying out batch data process in the most online stage that Fig. 8 provides for the specific embodiment of the invention
The schematic block diagram of specific embodiment four.
Detailed description of the invention
For making the purpose of the embodiment of the present invention, technical scheme and advantage clearer, below will with accompanying drawing and in detail
Narration clearly illustrates the spirit of disclosed content, and any skilled artisan is understanding present invention
After embodiment, when the technology that can be taught by present invention, being changed and modify, it is without departing from the essence of present invention
God and scope.
The schematic description and description of the present invention is used for explaining the present invention, but not as a limitation of the invention.
It addition, the element/component of same or like label is used to represent same or like portion used in drawings and the embodiments
Point.
About " first " used herein, " second " ... etc., censure order or the meaning of cis-position the most especially, also
It is not used to limit the present invention, its element described with constructed term only for difference or operation.
About direction used herein term, such as: upper and lower, left and right, front or rear etc., it is only the side with reference to accompanying drawing
To.Therefore, the direction term of use is used to illustrate not for limiting this creation.
About " comprising " used herein, " including ", " having ", " containing " etc., it is the term of opening, i.e.
Mean including but not limited to.
About used herein " and/or ", including the arbitrary of described things or all combine.
About term used herein " substantially ", " about " etc., in order to modify any can be with the quantity of microvariations or mistake
Difference, but this slight variations or error can't change its essence.It is said that in general, the microvariations modified of this type of term or error
Scope can be 20% in some embodiments, can be 10% in some embodiments, can be in some embodiments 5% or its
His numerical value.It will be understood by those skilled in the art that the aforementioned numerical value mentioned can adjust according to actual demand, be not limited thereto.
Some is in order to describe the word of the application by lower or discuss in the other places of this description, to provide art technology
Personnel are about guiding extra in the description of the present application.
A kind of method carrying out batch data process in the most online stage that Fig. 1 provides for the specific embodiment of the invention
The flow chart of specific embodiment one, please as it is shown in figure 1, receive in online transaction (online business) is mutual that batch service processes
Ask, on backstage, batch processing business datum is carried out data mart modeling, be not take up the time window of online transaction.
This detailed description of the invention shown in the drawings includes:
Step 101: process request in the online business middle batch service that receives alternately.Online business (on-line transaction) refers to connection
Reception and the process of machine service request are all real-time, process will be tied in the very first time after once online business has processed
Fruit passes to requesting party.Process request in the online business middle batch service that receives alternately, be not take up the time window of online business.
Step 102: the process request of described batch service is divided into process by the mark processing request according to described batch service
The queue that priority is different.In the specific embodiment of the present invention, queue includes very urgent queue, urgent queue and common team
Row, the processing priority of described very urgent queue, described urgent queue and described common queue reduces successively.Process completes non-
After batch service in the most urgent queue processes request, just can process the batch service in urgent queue and process request；Process
After becoming the batch service in urgent queue to process request, just can process the batch service in common queue and process request.The tightest
It is the highest that batch service in anxious queue processes request urgency level, and it can obtain the process of limit priority, its limit priority
Can maintain until it has processed, each batch service in the very urgent queue of entrance processes request and can obtain immediately always
Process, request is processed for the multiple batch services in very urgent queue, then according to the time order and function of request initiation time
Order processes successively.Batch service in urgent queue processes asks urgent journey to be less than at the batch service in very urgent queue
Reason request；And if the batch service in urgent queue process request can in default process time window threshold value (such as
Process time window threshold value could be arranged to 5 minutes) in completed, then this batch service process request can tie with normal process
Bundle；If it exceeds the process time window threshold value preset is still without having processed, then termination processes this batch service and processes request, and
And this batch service processes the priority asked and can decline, this batch service processes request and can be redistributed in common queue
Going, process by the process rule of common queue, the mode that this priority declines is consideration based on fairness, prevents long
Phase takies process resource, and is also to prevent from abusing very urgent queue, encourages to initiate short operation and processes request.Common queue
The urgency level that interior batch service processes request is minimum, only at the batch service in very urgent queue and urgent queue
After reason request is all processed, the batch service just starting to process in common queue processes request, most batch service
Process request belongs to common queue.
Step 103: sequentially process described batch during online business processes on backstage according to the priority of described queue
Business Processing is asked.In the detailed description of the invention of the present invention, step 103 specifically includes: step 1031 processes the phase at online business
Between process the described batch service in described very urgent queue when processing request, process sending out of request according to described batch service
The time of rising sequentially processes described batch service and processes request；Step 1032 processes described urgent queue during online business processes
When interior described batch service processes request, the initiation time processing request according to described batch service sequentially processes described batch
Business Processing is asked；Being fixing owing to each batch service processes the rule that processes of request, each batch service processes request
The input data object related to and output data object are also fixing, and difference simply processes varying in size, therefore of data volume
In the specific embodiment of the invention, step 1033 processes the described batch service in described common queue during online business processes
When processing request, process, according to described batch service, the handling capacity asked and handling duration processes described batch service successively and processes
Request.
See Fig. 1, in online transaction (online business) is mutual, receives batch service processes request, on backstage to batch industry
Business processes request and processes, and is not take up the time window of online transaction, and operator in T day (current trading day) with regard to energy
View batch service and process the result of request；The priority of request is processed in background process batch industry according to batch service
Business processes request, and processes the handling capacity of request and the other at background process lowest priority of handling duration according to batch service
Batch service processes request, it is possible to the most online window of Appropriate application so that each batch service processes request and has the opportunity to
To processing, operator just can view batch service at current trading day and process the result of request, improve expiring of client
Meaning degree.
A kind of method carrying out batch data process in the most online stage that Fig. 2 provides for the specific embodiment of the invention
The flow chart of specific embodiment two, as in figure 2 it is shown, setting processes time window threshold value, if batch service in urgent queue
The handling duration processing request exceedes process time window threshold value, then stop this batch service and process the process of request, and team
The process request of this batch service is included in common queue and is processed by row allocation unit, and in common queue, this batch service processes
Request continues the process of remaining data at termination.
In this detailed description of the invention shown in the drawings, step 1032 specifically includes:
Step 10321: process described batch service successively according to the sequencing of described request initiation time and process request.
In urgent inner queue, it is to carry out according to the rule of prerequisite variable that batch service processes the processing sequence of request.
Step 10322: judge that whether described batch service processes the process time span of request more than the first process time
Window threshold value.In the specific embodiment of the present invention, the first process time window threshold value is 5 minutes.
Step 10323: if it is greater, then termination processes this batch service processes request, and please by the process of this batch service
Ask and be included in common queue.If the handling duration of batch service process request exceedes process time window threshold in urgent queue
Value, then stop this batch service and process the process of request, and the process request of this batch service is included into general by queue assignment unit
Processing in logical queue, in common queue, this batch service processes request process (utilization of continuation remaining data at termination
Breakpoint mechanism supports that follow-up reentry processes, the record breakpoint information when stopping, heavy at breakpoint when reentry next time processes
Newly start to process).
Seeing Fig. 2, the mode that the priority that the specific embodiment of the invention provides declines is consideration based on fairness, prevents
Batch service in urgent queue processes request and takies process resource for a long time, and is prevented from abuse of emergency priority, encourages
Operator initiate the request of short job patch Business Processing.
A kind of method carrying out batch data process in the most online stage that Fig. 3 provides for the specific embodiment of the invention
The flow chart of specific embodiment three, as it is shown on figure 3, calculate in common queue at batch service according to handling capacity and handling duration
The changeable weight value of reason request, processes the batch service in common queue successively according to changeable weight value and processes request；At setting
Reason time window threshold value, if the handling duration of batch service process request exceedes process time window threshold value in common queue,
Then stop this batch service and process the process of request, and the request that this batch service processed is discharged to the tail of the queue of common queue, commonly
In queue, this batch service processes request process of continuation remaining data at termination.
In this detailed description of the invention shown in the drawings, step 1033 specifically includes:
Step 10331: calculate described batch service according to described handling capacity and described handling duration and process the dynamic of request
Weighted value.
Step 10332: process described batch service successively according to described changeable weight value and process request.
Step 10333: judge that whether described handling duration is more than the second process time window threshold value.
Step 10334: if it is greater, then termination processes this batch service processes request, and please by the process of this batch service
Seek the tail of the queue being discharged to common queue.
Seeing Fig. 3, the specific embodiment of the invention processes handling capacity and the process of request according to batch service in common queue
This batch service of duration calculation processes the changeable weight value of request, sequentially processes the batch in common queue according to changeable weight value
Business Processing is asked, and stops handling duration and exceed and process the batch service of time window threshold value and process request, and by this batch
Amount Business Processing request is discharged to the tail of the queue of common queue, prevents the batch service in common queue from processing request and takies process for a long time
Resource, encourages operator to initiate the request of short job patch Business Processing.
In the specific embodiment of the present invention, the computing formula of described changeable weight value Priority is:
Wherein, Priority represents the priority calculated, when its value is to process batch service process request in the past for n time
Handling capacity and the meansigma methods of handling duration ratio；ProcessTime (i) represents that processing batch service process in past i & lt asks
Ask the time used, calculate in units of millisecond；InputSize (i) represents that processing batch service process in past i & lt asks
Input the size of data when asking, calculate in units of byte；OutputSize (i) represents in past i & lt process batch industry
When business processes request, the size of output data, calculates in units of byte.
Wherein, the computing formula of described InputSize (i) and described OutputSize (i) is respectively as follows:
Wherein, InputRecordLength (k) is to process input data when batch service processes request in past i & lt
The record length of object；M represents that the number of input data object, such as i & lt process batch service and process request input data
Comprising m form, each form is an input data object；InputRecordCount (k) is to process in past i & lt to criticize
During amount Business Processing request, input data object is accessed the record strip number of process, and such as, each form has again 10000 row,
Every one record of behavior；K represents the order of input data object, the kth in i.e. m input data object；
OutputRecordLength (k) is that the record of output data object is long when past i & lt processes batch service process request
Degree；P represents the number exporting data object, and such as, i & lt processes batch service process request output packet and contains p form,
Each form is an output data object；OutputRecordCount (k) is to process batch service in past i & lt to process
Exporting data object during request and be accessed the record strip number of process, such as, each form has again 10000 row, every behavior one
Record；J represents the order of output data object, the jth in i.e. p output data object.
A kind of method carrying out batch data process in the most online stage that Fig. 4 provides for the specific embodiment of the invention
The flow chart of specific embodiment four, as shown in Figure 4, in order to prevent the batch service in common queue from processing and please rectificate further
Phase takies process resource, embodies and has equal opportunities and fairness, utilizes round policy control to process at the batch service of overlong time
Reason request, it is to take turns at which to process that round this batch service i.e. processes request, often carries out once reentry and processes, and wheel sub-value is just
From increasing by 1, wheel sub-value is the biggest, and processing priority is the lowest.
In this detailed description of the invention shown in the drawings, after step 10334, step 1033 also includes:
Step 10335: when termination processes the process request of this batch service, this batch service is processed the round that request is corresponding
Value adds 1, and wherein, the described priority of sub-value of taking turns is higher than described changeable weight value, and the described priority of sub-value of taking turns is less than institute
Stating mark, described to take turns the biggest priority of sub-value the lowest.
Seeing Fig. 4, the introducing of round strategy prevents the batch service process request in common queue from taking for a long time further
Processing resource, embody and have equal opportunities and fairness, the batch service utilizing round policy control to process overlong time processes request,
Operator are encouraged to initiate the longest business.
A kind of device carrying out batch data process in the most online stage that Fig. 5 provides for the specific embodiment of the invention
The schematic block diagram of specific embodiment one, device as shown in Figure 5 may apply in method as Figure 1-Figure 4, at connection
Receive batch service during machine affairs (online business) are mutual and process request, on backstage, batch processing business datum is carried out data and add
Work, is not take up the time window of online transaction.
In this detailed description of the invention shown in the drawings, the device carrying out batch data process in the most online stage includes connecing
Receive unit 10, queue assignment unit 20 and processing unit 30, wherein, receive unit 10 in online business middle reception batch alternately
Amount Business Processing request；Queue assignment unit 20 for processing the mark of request by described batch service according to described batch service
The request of process is divided into the queue that processing priority is different；Processing unit 30 is used for the priority according to described queue at online business
Sequentially process described batch service during process on backstage and process request.In the specific embodiment of the present invention, queue includes very
Urgent queue, urgent queue and common queue, described very urgent queue, described urgent queue and the process of described common queue
Priority reduces successively.Processing unit 30 farther includes: first processing module the 301, second processing module 302 and the 3rd processes
Module 303.Wherein, the institute in the first processing module 301 processes described very urgent queue at online business during processing
When stating batch service process request, the initiation time processing request according to described batch service sequentially processes at described batch service
Reason request.Second processing module 302 processes the described batch service in described urgent queue during processing at online business
When processing request, the initiation time processing request according to described batch service sequentially processes the process request of described batch service.The
The described batch service that three processing modules 303 process in described common queue during processing at online business processes request
Time, process, according to described batch service, the handling capacity asked and handling duration processes described batch service successively and processes request.
See Fig. 5, in online transaction (online business) is mutual, receives batch service processes request, on backstage to batch industry
Business processes request and processes, and is not take up the time window of online transaction, and operator in T day (current trading day) with regard to energy
View batch service and process the result of request；The priority of request is processed in background process batch industry according to batch service
Business processes request, and processes the handling capacity of request and the other at background process lowest priority of handling duration according to batch service
Batch service processes request, it is possible to the most online window of Appropriate application so that each batch service processes request and has the opportunity to
To processing, operator just can view batch service at current trading day and process the result of request, improve expiring of client
Meaning degree.
A kind of device carrying out batch data process in the most online stage that Fig. 6 provides for the specific embodiment of the invention
The schematic block diagram of specific embodiment two, as shown in Figure 6, setting processes time window threshold value, if batch industry in urgent queue
Business processes the handling duration of request and exceedes process time window threshold value, then stop this batch service and process the process of request, and
The process request of this batch service is included in common queue and is processed, in common queue at this batch service by queue assignment unit
Reason request continues the process of remaining data at termination.
In this detailed description of the invention shown in the drawings, described second processing module 302 farther includes the first process
Module 3021, first judge that submodule 3022 and first stops submodule 3023, wherein, the first processing module 301 for according to
The sequencing of request initiation time processes described batch service successively and processes request；First judge module 302 is used for judging institute
State batch service and whether process the process time span of request more than the first process time window threshold value；First stops module 303
When described process time span processes time window threshold value more than described first, termination processes this batch service and processes request,
And the process request of this batch service is included in common queue by described queue assignment unit 20.
Seeing Fig. 6, the mode that the priority that the specific embodiment of the invention provides declines is consideration based on fairness, prevents
Batch service in urgent queue processes request and takies process resource for a long time, and is prevented from abuse of emergency priority, encourages
Operator initiate the request of short job patch Business Processing.
A kind of device carrying out batch data process in the most online stage that Fig. 7 provides for the specific embodiment of the invention
The schematic block diagram of specific embodiment three, as it is shown in fig. 7, calculate in common queue according to described handling capacity and described handling duration
Batch service processes the changeable weight value of request, please according to the batch service process that changeable weight value processes in common queue successively
Ask；Setting processes time window threshold value, if the handling duration of batch service process request exceedes the process time in common queue
Window threshold value, then stop this batch service and process the process of request, and the process request of this batch service is discharged to common queue
Tail of the queue, in common queue, this batch service processes request process of continuation remaining data at termination.
In this detailed description of the invention shown in the drawings, described 3rd processing module 303 farther includes calculating sub module
3031, second process submodule 3032, second judge that submodule 3033 and second stops submodule 3034, wherein, calculating submodule
Block 3031 processes the changeable weight value of request for calculating described batch service according to described handling capacity and described handling duration；The
Two process submodule 3032 processes request for processing described batch service successively according to described changeable weight value；Second judges son
Module 3033 is used for judging that whether described handling duration is more than the second process time window threshold value；Second stops submodule 3034 uses
When described handling duration processes time window threshold value more than described second, termination processes this batch service and processes request, and should
Batch service processes request and is discharged to the tail of the queue of common queue.
Seeing Fig. 7, the specific embodiment of the invention processes handling capacity and the process of request according to batch service in common queue
This batch service of duration calculation processes the changeable weight value of request, sequentially processes the batch in common queue according to changeable weight value
Business Processing is asked, and stops handling duration and exceed and process the batch service of time window threshold value and process request, and by this batch
Amount Business Processing request is discharged to the tail of the queue of common queue, prevents the batch service in common queue from processing request and takies process for a long time
Resource, encourages operator to initiate the request of short job patch Business Processing.
In the specific embodiment of the present invention, the computing formula of described changeable weight value Priority is:
Wherein, Priority represents the priority calculated, when its value is to process batch service process request in the past for n time
Handling capacity and the meansigma methods of handling duration ratio；ProcessTime (i) represents that processing batch service process in past i & lt asks
Ask the time used, calculate in units of millisecond；InputSize (i) represents that processing batch service process in past i & lt asks
Input the size of data when asking, calculate in units of byte；OutputSize (i) represents in past i & lt process batch industry
When business processes request, the size of output data, calculates in units of byte.
Wherein, the computing formula of described InputSize (i) and described OutputSize (i) is respectively as follows:
Wherein, InputRecordLength (k) is to process input data when batch service processes request in past i & lt
The record length of object；M represents the number of input data object；InputRecordCount (k) is to process in past i & lt to criticize
During amount Business Processing request, input data object is accessed the record strip number of process；OutputRecordLength (k) is in mistake
I & lt is gone to process the record length of output data object when batch service processes request；P represents the number of output data object；
OutputRecordCount (k) is that output data object is accessed when past i & lt processes batch service process request to be processed
Record strip number.
A kind of device carrying out batch data process in the most online stage that Fig. 8 provides for the specific embodiment of the invention
The schematic block diagram of specific embodiment four, as shown in Figure 8, in order to prevent the batch service in common queue from processing request further
Take process resource for a long time, embody and have equal opportunities and fairness, utilize round policy control to process the batch service of overlong time
Processing request, it is to take turns at which to process that round this batch service i.e. processes request, often carries out once reentry and processes, takes turns sub-value
Just from increasing by 1, wheel sub-value is the biggest, and processing priority is the lowest.
In this detailed description of the invention shown in the drawings, described 3rd processing module 303 also includes arranging submodule 3035,
Wherein, submodule 3035 is set and for stopping to process when batch service processes request, this batch service is processed the wheel that request is corresponding
Sub-value adds 1, and wherein, the described priority of sub-value of taking turns is higher than described changeable weight value, and the described priority taking turns sub-value is less than
Described mark, described to take turns the biggest priority of sub-value the lowest.
See Fig. 8, present invention introduces round rule, it is possible to prevent the batch service in common queue from processing request further
Take process resource for a long time, embody and have equal opportunities and fairness, utilize round policy control to process the batch service of overlong time
Process request, encourage operator to initiate the longest business.
In the detailed description of the invention of the present invention, processing unit 30 also includes that module, bursting tube are had access in register module, inquiry
Control module, load balancing module, fractionation merge module and info push module etc..
The each batch service of register module registers processes request, and register information spinner to include the data type processed,
Data processing state, registers the information such as time, request teller (operator)；Teller is initiating batch service process request
Time can be according to actual business requirement designated treatment priority, the i.e. urgency level of this process: very urgent, urgent and common；Step on
Note book can preserve the processing priority that teller specifies, and rear extended meeting sends into different process queues according to these priority；Register
It is a table main, is the carrier of whole process data.
Security management and control module: for banking system, different human roles have different batch services to process request permissions.
When carrying out registering batch service process request or consulting the result of batch service process request, security management and control can be accessed
Module verifies the identity legitimacy of claimant, and batch service processes the reasonability of request, unsuccessfully or criticizes for authentication
When the scope of data of amount Business Processing request is illegal, carries out prompting and report an error.
Load balancing module: affect the process of on-line transaction in the daytime in order to prevent overload, devises a load all
Weighing apparatus module, load balancing module can arrange the maximum number of concurrent of process by parameter, can be temporary when reaching maximum number of concurrent
Time delay lag continuous batch service process request process, wait a present batch service process request be disposed
After, then carry out the process of new batch service process request；Introducing the stability of banking system and can of load balancing module
Control property is very important, and it is for provide operability in terms of Properties Control, it is to avoid occur in nothing in a helpless situation during bad performance
The difficult situation of plan.
Info push module: embody to strengthen user, makes user can understand the last state that request is processed in time,
Devise info push module.After each round process terminates, info push module obtains data process from register please
The relevant information asked for help, organizes a data process state information to be sent to this claimant.Implement and can pass through message team
The mode of row (MQ) sends message.
Split and merge module: this module provides user about the customizability of form size.For the form that some is bigger,
Number of pages is longer, and user's not aspect is had access to, and can split into many points of little forms by this mechanism and carry out successively having access to process；The most right
The scattered data result processed in many rounds can also be merged into a form by this mechanism.
Module is had access in inquiry: data processing request people can have access to module by this inquiry and have access to the result that request processes.
The same with register module, confidentiality based on bank data and the authority of user identity, module is had access in inquiry also must be through
Cross the checking of security management and control module, after being verified, just can be normally carried out having access to of data.When data are had access to, Gui Yuanke
With specify form represent form, such as every page of how many row information, if need split the latter merge represent, backstage is closed by fractionation
And the fractionation of module merges mechanism and specifically processes, i.e. provide a kind of inquiry side of having access to that can be customized according to client's needs
Formula, improves the motility that user is mutual, enhances user's service experience satisfaction.
The present invention provides a kind of method and device carrying out batch data process in the most online stage, in online transaction
Receive the process request of batch service during (online business) is mutual, on backstage, batch processing business datum carried out data mart modeling,
It is not take up the time window of online transaction, and operator just can view the place of batch service in T day (current trading day)
Reason result；Priority according to batch data business is in background process batch data business, and utilizes based on batch data business
The dynamic priority algorithm of handling capacity and the time of process is in background process lowest priority other batch data business, it is possible to rationally
Utilizing the most online window so that each batch data service request has the opportunity to be processed, operator are in current transaction
Day just can view the result of batch data business, improves the satisfaction of client.
The above-mentioned embodiment of the present invention can be implemented in various hardware, Software Coding or both combinations.Such as, this
Bright embodiment is alternatively and performs said method in data signal processor (Digital Signal Processor, DSP)
Program code.The present invention can also refer to computer processor, digital signal processor, microprocessor or field-programmable gate array
The several functions that row (Field Programmable Gate Array, FPGA) perform.Above-mentioned process can be configured according to the present invention
Device performs particular task, and it defines machine-readable software code or the firmware generation of the ad hoc approach that the present invention discloses by execution
Code completes.Software code or firmware code can be developed into different program languages and different forms or form.It is alternatively
Different target platform composing software codes.But, the software code performing task according to the present invention configures generation with other types
The different code pattern of code, type and language are without departing from spirit and scope of the invention.
The foregoing is only the schematic detailed description of the invention of the present invention, before without departing from the design of the present invention and principle
Put, equivalent variations that any those skilled in the art is made and amendment, the scope of protection of the invention all should be belonged to.
Claims (18)
1. the method carrying out batch data process in the most online stage, it is characterised in that the method includes:
Request is processed in the online business middle batch service that receives alternately；
It is different that the process request of described batch service is divided into processing priority by the mark according to the process request of described batch service
Queue；And
During online business processes, sequentially process described batch service on backstage according to the priority of described queue and process request.
2. the method carrying out batch data process in the most online stage as claimed in claim 1, it is characterised in that described team
Row include very urgent queue, urgent queue and common queue, described very urgent queue, described urgent queue and described commonly
The processing priority of queue reduces successively.
3. the method carrying out batch data process in the most online stage as claimed in claim 2, it is characterised in that according to institute
The priority stating queue sequentially processes the step of described batch service process request, specifically during online business processes on backstage
Including:
When processing the described batch service process request in described very urgent queue during online business processes, according to described
The initiation time of batch service process request sequentially processes described batch service and processes request.
4. the method carrying out batch data process in the most online stage as claimed in claim 2, it is characterised in that according to institute
The priority stating queue sequentially processes the step of described batch service process request, specifically during online business processes on backstage
Including:
When processing the described batch service process request in described urgent queue during online business processes, according to described batch
The initiation time of Business Processing request sequentially processes described batch service and processes request.
5. the method carrying out batch data process in the most online stage as claimed in claim 4, it is characterised in that according to institute
The initiation time stating batch service process request sequentially processes the step of described batch service process request, specifically includes:
Process described batch service successively according to the sequencing of request initiation time and process request；
Judge that whether described batch service processes the process time span of request more than the first process time window threshold value；And
If it is greater, then termination processes this batch service processes request, and common queue is included in the process request of this batch service
In.
6. the method carrying out batch data process in the most online stage as claimed in claim 2, it is characterised in that according to institute
The priority stating queue sequentially processes the step of described batch service process request, specifically during online business processes on backstage
Including:
When processing the described batch service process request in described common queue during online business processes, according to described batch
The handling capacity of Business Processing request and handling duration process described batch service successively and process request.
7. the method as claimed in claim 6 carrying out batch data process in the most online stage, it is characterised in that step
Suddenly, specifically include:
Calculate described batch service according to described handling capacity and described handling duration and process the changeable weight value of request；
Process described batch service successively according to described changeable weight value and process request；
Judge that whether described handling duration is more than the second process time window threshold value；And
If it is greater, then termination processes this batch service processes request, and the process request of this batch service is discharged to common queue
Tail of the queue.
8. the method carrying out batch data process in the most online stage as claimed in claim 7, it is characterised in that by this batch
After the step of the tail of the queue that amount Business Processing request is discharged to common queue, also include:
When termination processes the process request of this batch service, the wheel sub-value that this batch service is processed request corresponding adds 1, wherein, and institute
State the priority of wheel sub-value higher than described changeable weight value, and the described priority of sub-value of taking turns is less than described mark, described take turns
The biggest priority of sub-value is the lowest.
9. the method carrying out batch data process in the most online stage as claimed in claim 7, it is characterised in that described dynamic
The computing formula of state weighted value Priority is:
Wherein, Priority represents the priority calculated, and its value is to process for n time in the past when batch service processes request to handle up
Amount and the meansigma methods of handling duration ratio；ProcessTime (i) represents that processing batch service in past i & lt processes request institute
Time, in units of millisecond；InputSize (i) represents the input number when past i & lt processes batch service process request
According to size, in units of byte；OutputSize (i) represents the output when past i & lt processes batch service process request
The size of data, in units of byte.
10. the method carrying out batch data process in the most online stage as claimed in claim 9, it is characterised in that described
The computing formula of InputSize (i) and described OutputSize (i) is respectively as follows:
Wherein, InputRecordLength (k) is to process input data object when batch service processes request in past i & lt
Record length；M represents the number of input data object；InputRecordCount (k) is to process batch industry in past i & lt
When business processes request, input data object is accessed the record strip number of process；OutputRecordLength (k) is in the past i-th
The record length of output data object when secondary process batch service processes request；P represents the number of output data object；
OutputRecordCount (k) is that output data object is accessed when past i & lt processes batch service process request to be processed
Record strip number.
11. 1 kinds of devices carrying out batch data process in the most online stage, it is characterised in that this device includes:
Receive unit, for processing request in the online business middle batch service that receives alternately；
Queue assignment unit, is divided into the process request of described batch service for processing the mark of request according to described batch service
The queue that processing priority is different；And
Processing unit, for sequentially processing described batch according to the priority of described queue during online business processes on backstage
Business Processing is asked.
12. carry out the device of batch data process in the most online stage as claimed in claim 11, it is characterised in that described
Queue includes very urgent queue, urgent queue and common queue, described very urgent queue, described urgent queue and described general
The processing priority of logical queue reduces successively.
13. carry out the device of batch data process in the most online stage as claimed in claim 12, it is characterised in that described
Processing unit farther includes:
First processing module, processes at the described batch service in described very urgent queue during processing at online business
During reason request, the initiation time processing request according to described batch service sequentially processes the process request of described batch service.
14. carry out the device of batch data process in the most online stage as claimed in claim 12, it is characterised in that described
Processing unit farther includes:
Second processing module, processing the described batch service process in described urgent queue at online business during processing please
When asking, the initiation time processing request according to described batch service sequentially processes the process request of described batch service.
15. carry out the device of batch data process in the most online stage as claimed in claim 14, it is characterised in that described
Second processing module farther includes:
First processes submodule, please for processing the process of described batch service successively according to the sequencing of request initiation time
Ask；
First judges submodule, for judging that whether described batch service processes the process time span of request more than the first process
Time window threshold value；And
First stops submodule, and when described process time span processes time window threshold value more than described first, termination processes
This batch service processes request, and the process request of this batch service is included in common queue by described queue assignment unit.
16. carry out the device of batch data process in the most online stage as claimed in claim 12, it is characterised in that described
Processing unit farther includes:
3rd processing module, processing the described batch service process in described common queue at online business during processing please
When asking, process, according to described batch service, the handling capacity asked and handling duration processes described batch service successively and processes request.
17. carry out the device of batch data process in the most online stage as claimed in claim 16, it is characterised in that described
3rd processing module farther includes:
Calculating sub module, processes the dynamic of request for calculating described batch service according to described handling capacity and described handling duration
Weighted value；
Second processes submodule, processes request for processing described batch service successively according to described changeable weight value；
Second judges submodule, is used for judging that whether described handling duration is more than the second process time window threshold value；And
Second stops submodule, and when described handling duration processes time window threshold value more than described second, termination processes this batch
Amount Business Processing request, and this batch service is processed the tail of the queue asking to be discharged to common queue.
18. carry out the device of batch data process in the most online stage as claimed in claim 17, it is characterised in that described
3rd processing module also includes:
Submodule is set, for stopping to process when batch service processes request, this batch service is processed the wheel sub-value that request is corresponding
Adding 1, wherein, the described priority of sub-value of taking turns is higher than described changeable weight value, and the described priority of sub-value of taking turns is less than described
Mark, described to take turns the biggest priority of sub-value the lowest.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610439380.8A CN106095554B (en) | 2016-06-17 | 2016-06-17 | The method and device of batch data processing is carried out in the online stage in the daytime |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610439380.8A CN106095554B (en) | 2016-06-17 | 2016-06-17 | The method and device of batch data processing is carried out in the online stage in the daytime |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106095554A true CN106095554A (en) | 2016-11-09 |
| CN106095554B CN106095554B (en) | 2019-08-23 |
Family
ID=57236868
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610439380.8A Active CN106095554B (en) | 2016-06-17 | 2016-06-17 | The method and device of batch data processing is carried out in the online stage in the daytime |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106095554B (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106648874A (en) * | 2016-12-30 | 2017-05-10 | 中国建设银行股份有限公司 | Processing method and device for batched task |
| CN106685853A (en) * | 2016-11-23 | 2017-05-17 | 泰康保险集团股份有限公司 | Method and device for processing data |
| CN107819696A (en) * | 2017-11-22 | 2018-03-20 | 中国银行股份有限公司 | A kind of transaction flow control method and system |
| CN107832134A (en) * | 2017-11-24 | 2018-03-23 | 平安科技（深圳）有限公司 | multi-task processing method, application server and storage medium |
| CN109064311A (en) * | 2018-07-17 | 2018-12-21 | 中国银行股份有限公司 | Batch based on online mode is withholdd processing method and processing device |
| CN110333942A (en) * | 2019-07-05 | 2019-10-15 | 中国工商银行股份有限公司 | Electronic equipment and the task processing method executed by electronic equipment |
| CN110381150A (en) * | 2019-07-24 | 2019-10-25 | 中国工商银行股份有限公司 | Data processing method, device, electronic equipment and storage medium on block chain |
| CN110610414A (en) * | 2019-09-19 | 2019-12-24 | 中国银行股份有限公司 | Data processing method and system |
| CN111510396A (en) * | 2020-06-30 | 2020-08-07 | 腾讯科技（深圳）有限公司 | Message sending method and related device |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2002046887A2 (en) * | 2000-10-23 | 2002-06-13 | Xyron Corporation | Concurrent-multitasking processor |
| CN101246437A (en) * | 2008-01-28 | 2008-08-20 | 中兴通讯股份有限公司 | Built-in real-time system course equalization scheduling method |
| CN101261592A (en) * | 2007-03-07 | 2008-09-10 | 国际商业机器公司 | Method and apparatus for scheduling task requests |
| CN101510164A (en) * | 2009-03-25 | 2009-08-19 | 北京中星微电子有限公司 | Multi-service scheduling method, apparatus and system |
| CN102521055A (en) * | 2011-12-15 | 2012-06-27 | 广州杰赛科技股份有限公司 | Virtual machine resource allocating method and virtual machine resource allocating system |
| US20120297394A1 (en) * | 2011-05-19 | 2012-11-22 | International Business Machines Corporation | Lock control in multiple processor systems |
| CN104156264A (en) * | 2014-08-01 | 2014-11-19 | 西北工业大学 | Baseband signal processing task parallelism real-time scheduling method based on multiple GPUs |
| CN105389209A (en) * | 2015-12-25 | 2016-03-09 | 中国建设银行股份有限公司 | Asynchronous batch task processing method and system |
-
2016
- 2016-06-17 CN CN201610439380.8A patent/CN106095554B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2002046887A2 (en) * | 2000-10-23 | 2002-06-13 | Xyron Corporation | Concurrent-multitasking processor |
| CN101261592A (en) * | 2007-03-07 | 2008-09-10 | 国际商业机器公司 | Method and apparatus for scheduling task requests |
| CN101246437A (en) * | 2008-01-28 | 2008-08-20 | 中兴通讯股份有限公司 | Built-in real-time system course equalization scheduling method |
| CN101510164A (en) * | 2009-03-25 | 2009-08-19 | 北京中星微电子有限公司 | Multi-service scheduling method, apparatus and system |
| US20120297394A1 (en) * | 2011-05-19 | 2012-11-22 | International Business Machines Corporation | Lock control in multiple processor systems |
| CN102521055A (en) * | 2011-12-15 | 2012-06-27 | 广州杰赛科技股份有限公司 | Virtual machine resource allocating method and virtual machine resource allocating system |
| CN104156264A (en) * | 2014-08-01 | 2014-11-19 | 西北工业大学 | Baseband signal processing task parallelism real-time scheduling method based on multiple GPUs |
| CN105389209A (en) * | 2015-12-25 | 2016-03-09 | 中国建设银行股份有限公司 | Asynchronous batch task processing method and system |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106685853A (en) * | 2016-11-23 | 2017-05-17 | 泰康保险集团股份有限公司 | Method and device for processing data |
| CN106648874A (en) * | 2016-12-30 | 2017-05-10 | 中国建设银行股份有限公司 | Processing method and device for batched task |
| CN106648874B (en) * | 2016-12-30 | 2020-05-22 | 中国建设银行股份有限公司 | Processing method and device for batch tasks |
| CN107819696A (en) * | 2017-11-22 | 2018-03-20 | 中国银行股份有限公司 | A kind of transaction flow control method and system |
| CN107832134A (en) * | 2017-11-24 | 2018-03-23 | 平安科技（深圳）有限公司 | multi-task processing method, application server and storage medium |
| CN109064311A (en) * | 2018-07-17 | 2018-12-21 | 中国银行股份有限公司 | Batch based on online mode is withholdd processing method and processing device |
| CN110333942A (en) * | 2019-07-05 | 2019-10-15 | 中国工商银行股份有限公司 | Electronic equipment and the task processing method executed by electronic equipment |
| CN110381150A (en) * | 2019-07-24 | 2019-10-25 | 中国工商银行股份有限公司 | Data processing method, device, electronic equipment and storage medium on block chain |
| CN110381150B (en) * | 2019-07-24 | 2021-12-17 | 中国工商银行股份有限公司 | Data processing method and device on block chain, electronic equipment and storage medium |
| CN110610414A (en) * | 2019-09-19 | 2019-12-24 | 中国银行股份有限公司 | Data processing method and system |
| CN111510396A (en) * | 2020-06-30 | 2020-08-07 | 腾讯科技（深圳）有限公司 | Message sending method and related device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106095554B (en) | 2019-08-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106095554A (en) | The method and device of batch data process is carried out in the most online stage | |
| CN109783227A (en) | Method for allocating tasks, device, system and computer readable storage medium | |
| CN108197912A (en) | A kind of transaction processing system and method | |
| CN106817424A (en) | Method and system for controlling flowing of access | |
| Saini et al. | Lsp: Lightweight smart-contract-based transaction prioritization scheme for smart healthcare | |
| CN101483601A (en) | Scheduling method and apparatus based on service priority | |
| MX2010008166A (en) | Enrollment processing. | |
| CN107895324A (en) | Insurance examination ＆ verification apparatus and method | |
| US12164931B2 (en) | Ai intelligentialization based on signaling interaction | |
| CN105574977B (en) | credit card application method and system | |
| US20200320530A1 (en) | Maintenance plant management method, system and data management server | |
| CN113138847B (en) | Computer resource allocation scheduling method and device based on federal learning | |
| CN106327169A (en) | Electronic fund transferring method and apparatus | |
| CN109285069A (en) | Resource transfers method, apparatus and server | |
| CN113268336A (en) | Service acquisition method, device, equipment and readable medium | |
| CN106651372A (en) | Data processing method and system | |
| Tang et al. | Pricing model for dynamic resource overbooking in edge computing | |
| CN109508874A (en) | Mass of system analysis method, device, server and storage medium | |
| CN110009358A (en) | Resource allots risk prevention system method, apparatus and server | |
| CN109120548A (en) | A kind of flow control methods and device | |
| AU2015382255A1 (en) | Business coordination system and business coordination method | |
| CN113159751A (en) | Bank transfer amount sharing method and device | |
| CN112416558B (en) | Business data processing method, device and storage medium based on blockchain | |
| US20170200126A1 (en) | Retirement Score Calculator | |
| CN111311102A (en) | Resource ratio adjusting method, device, equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |