CN105488697A - Potential customer mining method based on customer behavior characteristics - Google Patents

Abstract

The invention provides a potential customer mining method based on customer behavior characteristics. The potential customer mining method comprises the following steps: a first step: data preprocessing; 1) data cleaning: deleting unnecessary recording rows at first; 2) forming a URL rule list; 3) user marking: accessing a log sheet to jointly identify the user through vinfo and client ID; 4) extracting features: with a session as a unit, analyzing the access source, the number of browsed pages, the number of browsed product detail pages, the number of browsed products, the page browsing duration, the product detail page browsing duration, the time of looking up a screening list, whether looking up a service topic and the first time browsing time period of the user in a single day of each user in single session, using whether the user is tend to purchase as a category attribute, and forming a training sample by the characteristic; 5) screening a training set; a second step: simplifying the characteristic attributes based on a rough set to improve the classification precision; and a third step: establishing a random forest potential customer identification model based on the customer behavior characteristics.

Description

A Potential Customer Mining Method Based on Customer Behavior Characteristics

technical field

The invention relates to the field of mining potential customers by analyzing website user access logs, in particular to a potential customer mining method based on customer behavior characteristics.

Background technique

At present, in the era of increasingly fierce market competition in the e-commerce era, by continuously expanding more new customers, effectively digging out potential customer groups from numerous browsers, and working hard to convert potential customers into actual customers, enterprises can gain more benefits and market Competitive Advantage. The purpose of potential customer mining is to provide accurate references and make corresponding decisions for the website to formulate corresponding service strategies.

The basic data of potential customer mining comes from the website's access log, which records the customer's visit behavior information when visiting a certain site, and this information is easy to obtain.

The access log records the visitor's IP, login ID, access time, VINFO, browsed product ID, REFERER (last visited page), REQUEST (visited page), SEARCH_WORD (search term), PROD_ID (browsed product), ORDER_ID and other information.

Table 1 Access log information

Iptonumber visitor ip Vinfo cookies Login_id login id Visit_time interview time request ask Referrer source User_agent machine used Prod_id product id Search_word search term Order_id order id Policy_id policy id User_agent machine id

Among them, REFERER and REQUEST are the most important information for analyzing the source of the visit, the destination of the visit, and judging whether the visitor intends to purchase, whether to add to the shopping cart, etc. Mining these behaviors extracts the behavioral characteristics of customers, which can more effectively reflect the category of customers, such as customers with which kind of visiting behavior characteristics are loyal customers, customers with which visiting behavior characteristics are pure visitors, Customers with which visit behavior characteristics are potential customers.

Therefore, how to find out the behavior characteristics of potential customers from the website access log is the problem that potential customers need to solve.

Contents of the invention

Purpose of the invention: The present invention provides a potential customer mining method based on customer behavior characteristics, formulates corresponding service strategies for websites, and provides accurate references and makes corresponding decisions.

Technical solution: In order to achieve the above object, the technical solution adopted in the present invention is: a potential customer mining method based on customer behavior characteristics, characterized in that:

Step 1: Data preprocessing;

Step1: Data cleaning

The original log records have accumulated a large amount of customer browsing information, many of which are redundant information not related to data mining, such as pictures, SMS verification, Logo pictures and other information. First, you need to delete unnecessary record lines;

Step2: Form a list of URL rules

Analyze the REQUEST field in the web data of the new station, and when it contains 'confirm', it represents the intention to purchase, etc., and finally forms a list of URL rules; when calculating features later, it is not necessary to analyze the request fields one by one, you can use the request field and the url rule list The url matches, get url_name;

Step3: User logo

In the access log table, users are jointly identified by vinfo, iptonumber, and customer ID (login_id);

Vinfo: Equivalent to cookie, which marks a computer;

iptonumber: ip address, the same computer will have different ip when logging in in different places;

login_id: member login id, login_id=-1 when non-member login.

Step4: Feature extraction

Taking a session as a unit, analyze the access source, number of pages browsed, number of product details pages browsed, number of products browsed, page browsing time, product details page browsing time, times of viewing the filter list, and whether to view the business of each user in a single session. Feature attributes such as topic, user's first browsing period in a single day, whether the user views the shopping cart, etc., and whether the user intends to purchase is used as the category attribute, and the training sample is finally formed with this feature;

Step5: Filter the training set

The behavior data information in the web log is the behavior information data generated by all users on the new website within a certain period of time, which includes people who have made many purchases (loyal customers) and people who have made few purchases (now There are customers), potential customers and behavioral data generated by people who have browsed the homepage of the website but have not browsed any products on the website (pure browsers);

By analyzing the number of purchases within a period of time, excluding customer data with multiple purchases, select customers who have purchased a product for the first time or customers who have not purchased after browsing as mining objects;

Step 2: Reduction of feature attributes based on rough sets;

For category attributes, some of the 11 feature attributes extracted in step 1 may be redundant. According to the rough set theory, the redundant attributes can be reduced without affecting the classification performance, so as to reduce the amount of calculation and improve the classification accuracy.

Method steps:

First use the relative positive domain to find the core:

Step1: Initialize data Core=φ, C={a ₁ ,a ₂ ,...,a _j }j=1,2,...11, a _j is the feature attribute, D={a ₁₂ } is the category attribute , calculate the relative positive domain POS _c (D);

Step2: B＝C-{a _j }, calculate the relative positive field POS _B (D), and compare POS _c (D) and POS _B (D). If POS _c (D)≠POS _B (D), then a _j is a core attribute, Core=Core∩B, loop to determine whether each attribute is a core attribute.

Step3: Return to Core, end.

Secondly, use attribute dependency to reduce Reduce:

Step1: Initialize data, remaining attributes RestAtt=C-Core, Reduce=Core,

Step2: Compare POS _core (D) and POS _c (D), if they are equal, then Core is reduction, otherwise go to step3;

Step3: Loop through each remaining attribute a _j in RestAtt, set Select the attribute a _k that makes the K value the largest, set Reduce=Reduce∪{a _k }, RestAtt=RestAtt-{a _k }, and compare POS _RestAtt (D) and POS _c (D), if they are equal, go to step4 , otherwise the loop continues.

Step4: Return to Reduce, end.

The Reduce at this time is the feature attribute of the final input classifier. Add a rough set based attribute filtering step.

Step 3: Random forest potential customer identification model based on customer behavior characteristics

The random forest algorithm is implemented using the language package randomForest4.6-6 of the R3.0.2 software. The program connects to the Oracle database through the data source ODBC, uses the function get_data() to obtain the required data, and uses the function cal_feture() to calculate the data features; screening training After the collection, call the random forest classification model model_rf to predict the feature data to obtain the potential customer ip and cookie information, and finally use the ip and cookie to find the potential customer user information in the existing data table and write it into the database.

In step three:

Step1: Connect to the database. The function of get_data() is to obtain the required data from the database. The parameter chan is the database connection, and cal_number is the date to obtain the data.

data=sqlQuery(chan,sql,stringsAsFactors=FALSE)

Use the odbcConnect() function in the RODBC package to establish a connection between R and the Oracle database:

chan＝odbcConnect("dm_xyz", uid＝'######', pwd＝'******')

Among them, the parameter dm_xyz is the system DSN name of the data source ODBC, uid is the user name, and pwd is the user login password;

After establishing the database connection, obtain the required data in the database by executing the sql statement; and add the data cleaning rule of step 1 in the sql statement;

Step2: URL matching, update browsing page information

Read the URL rule txt file in step 2 locally, match the keywords of the URL rule according to the REQUEST field in the data, update the VISIT_PAGE field, and set "-1" for records with no matching items.

Step3: Feature calculation cal_feature(data)

The function of the function cal_feature is to divide the single-day data obtained by the get_data function into different browsing sessions, calculate the features for a single session, and finally obtain the feature data set.

Step 4: Potential Customer Identification Model Performance Verification

Use oracle to edit the stored procedure to determine the proportion of potential customers who are dug out and actually buy within one month after the date of dug out, as a model performance verification indicator. If the customer's follow-up purchase rate predicted every day is relatively high and relatively stable, it proves that the model has better performance. Model performance indicators are given.

Beneficial effects of the present invention: the present invention provides a potential customer mining method based on customer behavior characteristics, uses a stable random forest algorithm to establish a classification model, has the characteristics of high efficiency and accurate data, and can provide accurate information for the website to formulate corresponding service strategies References and make corresponding decisions.

Description of drawings

Figure 1 is the structure diagram of the whole program, where model_rf.Rdata is the trained classification model.

detailed description

Step 1: Data preprocessing;

Step1: Data cleaning

The original log records have accumulated a large amount of customer browsing information, many of which are redundant information not related to data mining, such as pictures, SMS verification, Logo pictures and other information. First, unnecessary record lines need to be deleted.

The REQUEST of the row to be deleted is as follows

Step2: Form the applicant's new one-stop URL rule list

Analyze the REQUEST field in the applicant’s new web data, for example, analyze that when the request contains ‘baoxian’, it means viewing insurance topics, and when it includes ‘confirm’, it means that it intends to purchase, etc., and finally forms a list of new URL rules. In the subsequent calculation of features, it is not necessary to analyze the request fields one by one. The url_name can be obtained by matching the request field with the url in the url rule list, as shown in Table 2.

Table 2 New one-stop url rules

Step3: User logo

The access log table of the new site uses vinfo, iptonumber, and customer ID (login_id) to jointly identify users.

Vinfo: Equivalent to cookie, which marks a computer;

iptonumber: ip address, the same computer will have different ip when logging in in different places;

login_id: login id of a member of the new station, login_id=-1 when a non-member logs in.

Step4: Feature extraction

Taking a session as a unit, analyze the access source, number of pages viewed, number of product details pages viewed, number of products viewed, page browsing time, product details page browsing time, times of viewing the filter list, and whether to view insurance for each user in a single session. Feature attributes such as topic, whether to check the insurance dictionary, the time period of the user's first browsing in a single day, whether the user checks the shopping cart, etc., and whether the user intends to purchase is used as the category attribute to finally form a training sample.

a) Feature 1: Access source, according to the first referer of each session, distinguish the client's access source. Advertising, search engines, mail, direct access, others.

b) Feature 2: The number of pages browsed, the number of pages browsed by the user in a session.

c) Feature 3: The number of product details pages browsed, the number of product details pages browsed by users in a session.

d) Feature 4: page browsing time, the average browsing time of each page of a user in a session.

e) Feature 5: Browsing time of the product details page, the average browsing time of each product details page of the user in a session.

f) Feature 6: View the number of times to filter the list, and observe the number of times that ruquest has 'viewsearchlist' in a session.

g) Feature 7: Whether to view insurance topics. Check if there is 'baoxian' in ruquest.

h) Feature 8: Whether to check the insurance dictionary. Check if there is 'toptag' in ruquest.

i) Feature 9: Browsing period. A session first visit time visit_time

j) Feature 10: Yes No to check the shopping cart, yes is '1', no is '0'; whether to check the shopping cart is to observe whether there is shopping_car in ruquest.

k) Category attribute: whether it is intended to purchase, it is '1', and if it is not, it is '0'; whether it is intended to purchase is to judge whether there is a confirm in the ruquest, and the presence of confirm indicates the intention to purchase.

Step5: Filter the training set

The behavior data information in the web log is the behavior information data generated by all users on the new website within a certain period of time, which includes people who have made many purchases (loyal customers) and people who have made few purchases (now There are customers), potential customers, and people who have browsed the homepage of the website but have not browsed any products on the website (pure viewers).

If all the data is directly used to train the classification model, the result will have a large error. In order to avoid errors, the interference of other types of customer behaviors on the accuracy of the model should be excluded.

By analyzing the number of purchases within a period of time and excluding customer data of multiple purchases, the present invention selects customers who purchase a certain product for the first time or customers who have not purchased after browsing as mining objects.

Step6: Filter feature attributes

Through rough set theory, redundant attributes are eliminated, and the final characteristic attributes include access source, number of pages viewed, number of product details pages viewed, number of products viewed, page browsing time, product details page browsing time, number of searches, whether to view insurance topics , Whether to check the insurance dictionary, whether the user checks the shopping cart.

Step 3: Random forest potential customer identification model based on customer behavior characteristics

The random forest algorithm is implemented using the language package randomForest4.6-6 of the R3.0.2 software. The program connects to the Oracle database through the data source (ODBC), uses the function get_data() to obtain the required data, and uses the function cal_feture() to calculate the data features. After screening the training set, call the random forest classification model model_rf to predict the feature data to obtain potential customer ip and cookie information, and finally find potential customer user information in the existing data table through ip and cookie and write it into the database.

Step1: Connect to the database

The function of the function get_data() is to obtain the required data from the database, the parameter chan is the database connection, and the cal_number is the date to obtain the data.

data=sqlQuery(chan,sql,stringsAsFactors=FALSE)

Use the odbcConnect() function in the RODBC package to establish a connection between R and the Oracle database:

chan＝odbcConnect("dm_xyz", uid＝'######', pwd＝'******')

Among them, the parameter dm_xyz is the system DSN name of the data source (ODBC), uid is the user name, and pwd is the user login password.

After the database connection is established, the required data in the database is obtained by executing the sql statement. And add the data cleaning rule of step 1 in the sql statement.

Step2: URL matching, update browsing page information

Read the URL rule txt document in step 2 from the local, match the keywords of the URL rule according to the REQUEST field in the data, update the VISIT_PAGE field, and set "-1" for records with no matching items.

Step3: Feature calculation cal_feature(data)

The function of the function cal_feature is to divide the single-day data obtained by the get_data function into different browsing sessions, calculate the features for a single session, and finally obtain the feature data set.

(1) cookie information

(2) Whether to log in

(3) Number of pages viewed and average page browsing time

#Number of browsed pages

page_tag=which(tmp$VISIT_PAGE !='-1')

page_n=length(page_tag)

#Number of product details pages

prod_tag＝which(tmp$VISIT_PAGE＝＝'Product Details Page')

page_prod=length(prod_tag)

#Browsing time of pages and product details pages (page_n-1)

time=strptime(tmp[page_tag,2],"%Y-%m-%d%H:%M:%S")#Request time of the actual page

stay_time＝as.vector(diff(time))#page request time difference as the stay time of the page

page_time_avg＝sum(stay_time,na.rm＝T)/page_n#Average page access time

prod_time＝stay_time[which(page_tag%in%prod_tag)]#The total stay time of the product details page

prod_time_avg＝sum(prod_time,na.rm＝T)/page_prod#Average duration of product details page

(4) Number of times the filter list was viewed

#Filter list times (0 means no search)

sear_n=length(which(tmp$VISIT_PAGE=="search list"))

(5) Have you checked the shopping cart

(6) Have you checked the topic of insurance?

(7) Have you checked the insurance dictionary

(8) The type of the source website, judge the source website according to the keywords in the REFERER and REFERER_SOURCE_WORD fields.

(9) Intention to purchase (predictor variable)

Finally, an obs observation record is synthesized and returned, and users who purchase for the first time and users who have not purchased after browsing are screened to form a training sample.

Step4: Update the potential customer information table data update_info(channe,cal_number) in the database

This function is the most important function of the whole program, which is used to predict potential customers, obtain potential customer information and write it into the database. The get_data() function and cal_feature() function will be called in the function.

(1) Obtain data, calculate features and predict potential customers

data=get_data(channel,cal_number)#Get the data of the date corresponding to cal_number

feature=cal_feature(data)#Calculate the feature data corresponding to the daily data

feature[,14]=as.factor(feature[,14])#Set the buy_flag field to factor

feature0=feature[which(feature$buy_flag==0),]#Extract the record marked as 0 as the predicted object

pre_rf＝predict(model_rf,feature0[,3:14])#Use the model to predict

potential_ip＝feature0[which(pre_rf＝＝1),1:2]#Take the ip and cookie marked as 1 as a potential customer

(2) Through the login_id, associate the user_info table and update the basic information of the user

The basic information includes the user's age, gender, email address, birthday and other information, so that the website can do some offline promotion strategies.

Step5: Start the program to automatically identify potential customers

When executing the whole program, you only need to execute the update_info(channe, cal_number) function, get the system date before executing the function, and push the system date forward one day as the cal_number parameter when the program is executed.

Step6: Model performance verification

The finally dug out potential customer information table is associated with the new station log table through vinfo, and the visit behavior of each potential customer 30 days after the digging date is obtained. If the customer has a payment success flag within the next 30 days, that is, the request contains 'paysuccess', and the order ID in the log table of the new station intercepted by the regular expression is paid in the order information table, the payment status is considered to be paid at this time. Potential customers did subsequently purchase behavior, proving that the model successfully predicted. Therefore, the follow-up purchase rate is used as the model performance verification index.

It should be pointed out that, without departing from the principle of the present invention, appropriate modifications or substitutions should be made, and these modifications or substitutions should also be regarded as the protection scope of the present invention.

Claims (3)

1., based on potential customers' method for digging of customer action feature, it is characterized in that:

Step one: data prediction;

1): data cleansing

Original log record have accumulated a large amount of client's browsing informations, is much the redundant information irrelevant with data mining, such as picture, short-message verification, Logo pictorial information, first needs to delete unwanted record row;

2): form URL list of rules

Analyze the REQUEST field in a new station web data, to when comprising ' confirm ', represent purpose purchase etc., final formation URL list of rules; During subsequent calculations feature, do not need to analyze request field one by one, according to request field with the url coupling in url list of rules, url_name can be obtained;

3): user label

Combine with vinfo, iptonumber, Customer ID to identify user in access log table;

Vinfo: be equivalent to cookie, indicates a computing machine;

Iptonumber:ip address, same computer logs in different place, has different ip;

Login_id: Member Entrance id, login_id=-1 when non-member logs in.

4): feature extraction

In units of a session, analyze the access source of each user in single session, browsing pages number, browse product details number of pages, browse product number, page browsing duration, product details page browsing duration, check filtered list number of times, whether check business topic, user's odd-numbered day browses the period first, whether user checks the characteristic attributes such as shopping cart, and using user, whether purpose is bought as category attribute, finally forms training sample with this feature;

5): screening training set

Behavioral data information in web daily record is the behavioural information data that in certain time period, total user produces on a new website, station, this wherein just includes the people and loyalty customer that repeatedly do shopping, the people that shopping number of times is few and existing customer, potential customers and browsed site home page, but the behavioral data not browsing that the people of commodity in any website and pure viewer produce;

By analyzing the purchase number of times in a period of time, excluding the customer data repeatedly bought, choosing the client that initial purchase is carried out to a certain product or the client do not bought after browsing as excavation object;

Step 2: based on the Characteristic Attribute Reduction of rough set;

For category attribute, 11 characteristic attributes that step one is extracted, redundant attributes according to rough set theory, under the prerequisite not affecting classification performance, must be divided out by the characteristic attribute of redundancy, thus reduces operand, improves nicety of grading;

Method step:

First relative positive field is utilized to ask core Core:

1): initialization data Core=φ, C={a ₁, a ₂..., a _jj=1,2 ... 11, a _jfor characteristic attribute, D={a ₁₂be category attribute, calculate relative positive field POS _c(D);

2): B=C-{a _j, calculate relative positive field POS _b, and compare POS (D) _c(D), POS _b(D).If POS _c(D) ≠ POS _b(D), then a _jfor core attributes, Core=Core ∩ B, whether each attribute of cycle criterion is core attributes;

3): return Core, terminate;

Next utilizes attribute dependability to ask yojan Reduce:

1): initialization data, residue attribute RestAtt=C-Core, Reduce=Core,

2): compare POS _core(D), POS _c(D), if equal, then Core is yojan, otherwise forwards step3 to;

3): each residue attribute a in circulation RestAtt _jif, select the attribute a making K value maximum _k, make Reduce=Reduce ∪ { a _k, RestAtt=RestAtt-{a _k, and compare POS _restAttand POS (D) _c(D), if equal, forward step4 to, otherwise continue circulation;

4): return Reduce, terminate;

Reduce is now the characteristic attribute of final input sorter;

Step 3: based on the random forest potential customers model of cognition of customer action feature;

Random forests algorithm uses the lingware bag randomForest4.6-6 of R3.0.2 software to realize, program connects oracle database by data source ODBC, use function get_data () to obtain desired data, use function cal_feture () to calculate data characteristics; After screening training set, call random forest disaggregated model model_rf and prediction is carried out to characteristic obtain potential customers ip and cookie information, finally by ip and cookie search in data with existing table potential customers user profile and in write into Databasce;

In step 3:

1): connection data storehouse, the function of function get_data () for obtain desired data from database, and parameter chan is DataBase combining, and cal_number is the required date obtaining data,

data＝sqlQuery(chan,sql,stringsAsFactors＝FALSE)

In being wrapped by RODBC, odbcConnect () function is set up R and is connected with oracle database:

chan＝odbcConnect("dm_xyz",uid＝'######',pwd＝'******')

Wherein, parameter d m_xyz is the system DSN name of data source ODBC, and uid is user name, and pwd is user login code;

After building database connects, obtain desired data in database by performing sql statement; And in sql statement, add the data cleansing rule of step one;

2): URL mates, browsing pages information is upgraded;

Read in the URL rule txt document of step 2 by this locality, according to the key word of the REQUEST fields match URL rule in data, upgrade VISIT_PAGE field, the record without occurrence is then set to "-1 ";

Step3: feature calculation cal_feature (data)

The function of function cal_feature is the odd-numbered day Data Segmentation that get_data function obtains is become different to browse session, calculates feature finally obtain characteristic data set to single session;

Step 4: potential customers' model of cognition performance verification

Utilizing oracle edit and storage process, judging that the potential customers that dig out are digging out the real ratio bought in day after date one month, as model performance checking index.

2. potential customers' method for digging according to claim 1, is characterized in that, in units of a session, is analyzed as follows feature, finally forms training sample:

Feature 1: access source, according to first referer of each session, distinguishes the access source of client; Advertisement putting, search engine, mail, directly access, other;

Feature 2: browsing pages number, gets the page number that in a session, user browses;

Feature 3: browse product details number of pages, gets user in a session and browses product details number of pages;

Feature 4: page browsing duration, in a session, the duration of browsing of each page of user is averaged; ;

Feature 5: product details page browsing duration, in a session, the duration of browsing of each product details page of user is averaged;

Feature 6: the number of times checking filtered list, observing ruquest in a session has ' number of times of viewsearchlist ';

Feature 7: whether check insurance topic; Observe in ruquest and whether have ' baoxian ';

Feature 8: whether check insurance dictionary; Observe in ruquest and whether have ' toptag ';

Feature 9: browse the period; A session access time visit_time first

Feature 10: whether check shopping cart, be ' 1 ', no is ' 0 '; Whether check shopping cart observes in ruquest whether have shopping_car;

Category attribute: whether purpose is bought, and be ' 1 ', no is ' 0 '; Whether purpose is bought is judge whether have confirm in ruquest, containing confirm, the buying behavior of generation purpose is described.

3. potential customers' method for digging according to claim 1, is characterized in that, the feature extraction of data processing is:

(1) cookie information comprises

It is not the record value of '-1 ' that #cookie information gets VINFO field first

(2) whether log in

(3) browsing pages number and the page on average browse duration

# browsing pages number

page_tag＝which(tmp$VISIT_PAGE！＝'-1')

page_n＝length(page_tag)

# product details page number of pages

Prod_tag=which (tmp $ VISIT_PAGE==' product details page ')

page_prod＝length(prod_tag)

The # page and product details page browse duration (page_n-1)

The request time of time=strptime (tmp [page_tag, 2], " %Y-%m-%d%H:%M:%S ") # actual pages

Stay_time=as.vector (diff (time)) the # page request mistiming is as the residence time of the page

Page_time_avg=sum (stay_time, na.rm=T)/page_n# page average access duration

Prod_time=stay_time [which (page_tag%in%prod_tag)] # product details page total residence time

Prod_time_avg=sum (prod_time, na.rm=T)/average duration of page_prod# product details page

(4) number of times of filtered list is checked

# filtered list number of times (0 expression is not searched for)

Sear_n=length (which (tmp $ VISIT_PAGE==" search listing "))

(5) whether shopping cart was checked

(6) insurance topic whether was checked

(7) insurance dictionary whether was checked

(8) type of source web, judges source web according to REFERER and REFERER_SOURCE_WORD field keys.

(9) time period of browsing, the different browsing time section of user is distinguished according to VISIT_TIME field.

(10) whether purpose purchase and predictive variable

Final synthesis obs observational record returns, and screening first purchase and the user do not bought after browsing, form training sample.