Extract data via JSON API and make reports
$1500-3000 USD
Paid on delivery
We need to build a robust web application in PHP and mysql that will do the following:
1.- Data extraction and saving:
1A.- Connect via JSON API to a popular website that lists products in a category to sell and extract several variables.
1A-1.- Connect to an URL in the format: www.URL.com/category=CAT1&limit=200&offset=0 where CAT1 is dynamic category code of products(to be explained later on this document), limit is the allowed max products to display per URL request, offset is the start of the listing or paging.
1A-2.- Extract and save several variables from each row of the product listing to the database: ProductID, ITEM-NAME, Price, QTY-SOLD, location, SellerID, Product-URL. 200 products info should be saved here.
1A-3.- Connect to the same URL now with Offset=201 and repeat steps X amount of times to be defined later on.
1A-4.- When X amount of times has been repeated on the same category, step 1 will be repeated now with a new category. Steps 1 to 4 will be repeated until all categories with x amount of times are saved.
1B.- Connect via JSON API to the seller page to extract the Seller Name
1B-1.- IF NickName of a SellerID is not known, then:
1B-2.- Connect to URL [login to view URL] and extract and save variable NICKNAME.
2.- Assign Internal-ID:
2A.- All item-ID should be assigned an internal-ID. When an itemID expires, it is republished with a new itemID. We need to track and know that a new itemid is in fact just the old item republished. With the parent-item-id variable we can know this information.
2A-1.- When a new ITEMID is detected, connect to item URL [login to view URL] and look for parent_item_id and store variable.
2A-2.- If parent_item_id is equal to another itemID then assign Internal-ID of previous ITEMID, else assign new internal-ID.
2B.- We need to adjust for QTY-SOLD from the old to new itemID when a new itemID is assigned to an existing internal-ID
2B-1.- Connect to the old item ID page. [login to view URL] where ITEMID is the old item ID.
2B-2.- Store the same variables as in step 1A-2
Note: What we are doing here is the following: Suppose we recorded the variables of ITEMID1 on June 15. The ITEMID someone expired on June17 and we detected the republished new ITEMD2 on June 20. We need to know the item QTY-SOLD on June 17 since its an unknown.
Note2: all of process numer 2 should only be done after the initial loading or starting phase. It is not needed to run on the first time.
3.- Data extraction Screen and automation.
A.- Thread Status list
3A-1.- There should be a screen where the status and URL of the current data extraction thread is displayed. There should be an option to pause, stop, start the thread. Diferent URL types should be on different threads like categories or sellerid or itemid URLs.
3A-2.- The app should have the ability to configure and run multiple threads of data extraction and also the ability to throttle them by amount of URLs per minute per thread.
3A-3.- The app should be able to be run manually or automated by X amount of time like once a week, once a month, etc. by any method desired either crons or another method.
3A-4.- Particular attention should be devoted to anticipate any errors and display them accordingly to take corrective action.
There are additional points, please look at attached document. Thanks
Basically this is a PHP app with mysql that connects via json api to a popular site
Time frame:
1 month
Project ID: #7802076
About the project
35 freelancers are bidding on average $3343 for this job
Greetings, I have read Project Description & fully understood your requirements but I am afraid that time-frame of 1 month and your budget is low, so would you be flexible in it? Please drop me a message thru' PMB t More
Hi, We have reviewed your requirements for building this multi threaded automated system. We are interested coz we have worked on such data extraction solutions and automation. Thank You
Hello, I have gone through the attached doc properly. Are you adamant on PHP to be used? As PHP does not support multi threading in true sense I would recommend .NET Can we use MongoDB for increased performanc More
I can do this, no problem. I'll dive deeper in your doc file (don't have ms word right now) and will be ready to discuss this tomorrow. Thanks .
Hello - Nice to see your project posted here. I want to know if it will be an automated process linked to some cron job? Thanks.
Hi, seems a great project to work on. We have gone through the brief and will be great to discuss the project and this proposal further with you. We are a web development company based in India. We do mostly Open Sourc More
Hey hi I am Varun, we are experts in JSON. I can assure you that we can complete this project for you. Please initiate a personal message to discuss further.
Hi, I read all your requirements and understand what you need. I am very familiar with JSON API because I've made and used JSON API for my previous mobile app. Frankly I am very good in web service development and co More
Hi, my name is Awadesh. I am. Dr versed with API and neon and can do this task. Let's discuss more
