Pan Fashion Home > Who Knows > How to get Awins Create-a-feed products into your mysql database

How to get Awins Create-a-feed products into your mysql database


Awins new Create-a-feed system is literally the bees knees, but, how do you get the data into a mysql database so you can begin building that website you’ve dreamed of building, the answer is here.
Make sure your create-a-feed interface looks like this

awins create-a-feed

awins create-a-feed

Copy the code below into a php file, read the comments within the file and you should be good to go

<?
//download and unpack gzipped csv file from awin

// maximum execution time in seconds
set_time_limit(120);
//set memory limit
ini_set("memory_limit","3072M");

// insert your download url below from awins create-a-feed system
$url = "YOUR_DOWNLOAD_URL_FROM_AWIN_CREATE_A_FEED";  //make sure you've checked the csv option and the gzip option in the create-a-feed interface.
$filename = "download.csv.gz";
$newfname = $filename;

$file = fopen ($url, "rb");
if ($file) {
  $newf = fopen ($newfname, "wb");

  if ($newf)
  while(!feof($file)) {
    fwrite($newf, fread($file, 1024 ), 1024 );
  }
}

if ($file) {
  fclose($file);
}

if ($newf) {
  fclose($newf);
}

$filename =  'download.csv.gz';
$zd = gzopen($filename, "r");
$contents = gzread($zd, 10000000000); //because I have no idea how big your file will be
gzclose($zd);

$filename='awin1.csv';
$tempvar = fopen($filename,"w");
fwrite($tempvar, $contents);
fclose($tempvar);

?>

<?php
//set up dbase connection and open dbase
$dbhost = 'DBASE_HOST'; //normally localhost
$dbuser = 'DBASE_USERNAME'; //your dbase username
$dbpass = 'DBASE_PASSWORD'; //your dbase password
$conn = mysql_connect($dbhost, $dbuser, $dbpass) or die('Error connecting to mysql');
$dbname = 'DBASE_NAME'; //your dbase name
$table_name = 'test'; //change this to whatever you want your table to be called
mysql_select_db($dbname);
?>

<?
//create the table if it isn't there already.
$query = 'CREATE TABLE IF NOT EXISTS `'.$table_name.'` ('.
  '`merchant_id` double default NULL,'.
  '`merchant_name` char(255) default NULL,'.
  '`aw_product_id` double default NULL,'.
  '`merchant_product_id` char(255) default NULL,'.
  '`upc` double default NULL,'.
  '`ean` double default NULL,'.
  '`mpn` char(255) default NULL,'.
  '`isbn` double default NULL,'.
  '`model_number` char(255) default NULL,'.
  '`product_name` char(255) default NULL,'.
  '`description` longtext,'.
  '`specifications` char(255) default NULL,'.
  '`promotional_text` char(255) default NULL,'.
  '`merchant_category` char(255) default NULL,'.
  '`category_id` double default NULL,'.
  '`category_name` char(255) default NULL,'.
  '`language` char(255) default NULL,'.
  '`brand_name` char(255) default NULL,'.
  '`merchant_deep_link` char(255) default NULL,'.
  '`merchant_thumb_url` char(255) default NULL,'.
  '`merchant_image_url` char(255) default NULL,'.
  '`aw_deep_link` char(255) default NULL,'.
  '`aw_thumb_url` char(255) default NULL,'.
  '`aw_image_url` char(255) default NULL,'.
  '`delivery_time` char(255) default NULL,'.
  '`valid_from` char(255) default NULL,'.
  '`valid_to` char(255) default NULL,'.
  '`currency` char(255) default NULL,'.
  '`search_price` double default NULL,'.
  '`store_price` double default NULL,'.
  '`rrp_price` double default NULL,'.
  '`display_price` double default NULL,'.
  '`delivery_cost` double default NULL,'.
  '`web_offer` double default NULL,'.
  '`pre_order` double default NULL,'.
  '`in_stock` double default NULL,'.
  '`stock_quantity` double default NULL,'.
  '`is_for_sale` double default NULL,'.
  '`warranty` char(255) default NULL,'.
  '`condition` char(255) default NULL,'.
  '`product_type` char(255) default NULL,'.
  '`parent_product_id` double default NULL,'.
  '`commission_group` char(255) default NULL,'.
  'KEY `aw_product_id` (`aw_product_id`),'.
  'KEY `category_id` (`category_id`),'.
  'KEY `merchant_id` (`merchant_id`),'.
  'KEY `merchant_product_id` (`merchant_product_id`),'.
 'KEY `parent_product_id` (`parent_product_id`)'.
') ENGINE=MyISAM DEFAULT CHARSET=latin1;';

$result = mysql_query($query);
?>

<?
//load the data from the csv file into the database
$query = 'LOAD DATA LOCAL INFILE  \'awin1.csv\' INTO TABLE '.$table_name.' FIELDS TERMINATED BY \',\' ENCLOSED BY \'"\' LINES TERMINATED BY \'\n\' IGNORE 1 LINES';
$result = mysql_query($query);
?>

<?
//close dbase connection
mysql_close($conn);
?>

You should of course have a database set up already :)

Hmm, thats got me thinking, how about an awin create-a-feed direct to wordpress posts…??? watch this space!!!

Coming soon, the code to turn this database into some kind of webpage, mmm, can’t wait.

  1. #1 by Mike at June 10th, 2009

    Warning: fopen(download.csv.gz) [function.fopen]: failed to open stream: No such file or directory.

    I get that error on this line:

    $newf = fopen ($newfname, “wb”);

    Any ideas? Very useful script if I can get it to work! :)

  2. #2 by admin at June 10th, 2009

    Mike :

    Warning: fopen(download.csv.gz) [function.fopen]: failed to open stream: No such file or directory.

    I get that error on this line:

    $newf = fopen ($newfname, “wb”);

    Any ideas? Very useful script if I can get it to work! :)

    It would appear that the file isn’t downloading. Check in the folder on your server where the script is and see if there is a ‘download.csv.gz’ file there. The script is reporting to you that it isn’t there.

    Make sure the download url is correct, try it in a browser address bar to make sure it retrieves the file, if it does, then your server doesn’t allow remote fopen. Something you will need to take up with your hosting provider.

  3. #3 by James at June 12th, 2009

    Very interesting article! :) I’m just wondering if you can apply the same principle with data feeds where you have custom/additional fields for certain merchants.. any ideas?

  4. #4 by admin at June 12th, 2009

    James :

    Very interesting article! :) I’m just wondering if you can apply the same principle with data feeds where you have custom/additional fields for certain merchants.. any ideas?

    You can apply the principle to any gzipped csv, the only changes to be made would be the creation of the database, it would need to reflect the structure of your datafeed.

    If you already have a database table of the same structure, then it wouldn’t be an issue, as the lines

    $query = ‘LOAD DATA LOCAL INFILE \’awin1.csv\’ INTO TABLE ‘.$table_name.’ FIELDS TERMINATED BY \’,\’ ENCLOSED BY \’”\’ LINES TERMINATED BY \’\n\’ IGNORE 1 LINES’;
    $result = mysql_query($query);

    just dump the entire csv minus the top line (normally the field names) into the table.

  5. #5 by James at June 12th, 2009

    ahh.. that got me thinking for 2 possible solutions

    1/ One solution is get the field name from the CSV and dynamically create the table based on the csv fieldnames.

    2/ Or check if fieldnames exists in table from csv fields; if not alter table to add fields in table then do your query with “TERMINATED BY \’\n\’ IGNORE 1 LINES (field1, field2, field3, …. , fieldn)

    :)

  6. #6 by admin at June 12th, 2009

    If we create the fields from the first line of the csv, then the field definition would be a bit hit and miss, how would we know which field was a double, or longtext or varchar?

    Perhaps, read the first two lines, one for the field names, one for a best-guess solution to the field type?

  7. #7 by admin at June 12th, 2009

    James :

    ahh.. that got me thinking for 2 possible solutions
    1/ One solution is get the field name from the CSV and dynamically create the table based on the csv fieldnames.
    2/ Or check if fieldnames exists in table from csv fields; if not alter table to add fields in table then do your query with “TERMINATED BY \’\n\’ IGNORE 1 LINES (field1, field2, field3, …. , fieldn)

    Let me point you at this page James

    http://www.panfashion.co.uk/06/how-to-create-a-mysql-table-from-a-csv-file-with-unknown-fields/

  8. #8 by Zoran at June 23rd, 2009

    Hi, Everything dynamic and very positively! :)
    Have a nice day
    Zoran

  9. #9 by blogman at July 11th, 2009

    I’m having some trouble with this code. Everything seems to work fine until it has to LOAD DATA LOCAL INFILE \’awin1.csv\’ INTO TABLE.

    I’m ending up with empty mysql tables but with all the product fields created.

    :P

  10. #10 by admin at July 13th, 2009

    blogman :

    I’m having some trouble with this code. Everything seems to work fine until it has to LOAD DATA LOCAL INFILE \’awin1.csv\’ INTO TABLE.

    I’m ending up with empty mysql tables but with all the product fields created.

    :P

    Perhaps your mysql version along with your server permissions don’t allow for the ‘LOAD DATA LOCAL INFILE’ command

  11. #11 by Quentin at August 25th, 2009

    My hosting won’t allow remote fopen, any chance of replacing this part of the script with Curl?

  12. #12 by Brett at December 8th, 2009

    Hi im trying to get your code working, but im having the same trouble that the table is created with all of the fields but no data is added. Does this section of code change whatever the csv file is called in the zip file and rename is awin1.csv?

    $filename = ‘download.csv.gz’;
    33.$zd = gzopen($filename, “r”);
    34.$contents = gzread($zd, 10000000000); //because I have no idea how big your file will be
    35.gzclose($zd);
    36.
    37.$filename=’awin1.csv’;
    38.$tempvar = fopen($filename,”w”);
    39.fwrite($tempvar, $contents);
    40.fclose($tempvar);

  13. #13 by admin at December 8th, 2009

    Hi Brett, have a look at the comments on this post, in particular, the second to last comment LINK

  14. #14 by Brett at December 8th, 2009

    Hi thanks for your reply. Ive tried that but still seems to have the same trouble. Its even started kicking up Server 500 Internal Errrors now which i cant work out.

    I tried the other code from the other post but get the following message:

    CREATE TABLE IF NOT EXISTS anything ( merchant_id,merchant_name,aw_product_id,merchant_product_id,upc,ean,mpn,isbn,model_number,product_name,description,specifications,promotional_text,merchant_category,category_id,category_name,language,brand_name,merchant_deep_link,merchant_thumb_url,merchant_image_url,aw_deep_link,aw_thumb_url,aw_image_url,delivery_time,valid_from,valid_to,currency,search_price,store_price,rrp_price,display_price,delivery_cost,web_offer,pre_order,in_stock,stock_quantity,is_for_sale,warranty,condition1,product_type,parent_product_id,commission_group longtext ) Table ‘web180-data.anything’ doesn’t exist

    Do i need to change any of line 114 or just lines 74 and 76? (I have already changed both to us \t as mentioned for comma seperated files. Thanks for yout time.

  15. #15 by Brett at December 9th, 2009

    Sorry ignore that i read the comments about changing the lines wrong, its a comma seperated csv so nothing needs changing suprise suprise!

    I do still get the:

    CREATE TABLE IF NOT EXISTS anything (tables field names) Table ‘web180-data.anything’ doesn’t exist

    which i cant seem to stop.

  16. #16 by admin at December 9th, 2009

    Brett, it seems that this problem is fairly common, and, I guess it’s down to the system mysql user not having ‘create table’ rights.

    I would fire up your phpmyadmin from your hosting control panel, then dump the sql into the web180-data database SQL box, then, your table will be there, and you can import with impunity.

    Hope this helps

  17. #17 by Brett at December 9th, 2009

    Should that just be a case of copting the following into the SQL query box?

    CREATE TABLE IF NOT EXISTS anything ( merchant_id int(11), merchant_name char(255), aw_product_id int(11), merchant_product_id int(11), upc int(11), ean int(11), mpn char(255), isbn int(11), model_number char(255), product_name char(255), description char(255), specifications char(255), promotional_text char(255), merchant_category char(80), category_id char(80), category_name char(255), language char(255), brand_name int(11), merchant_deep_link char(255), merchant_thumb_url char(255), merchant_image_url char(255), aw_deep_link char(255), aw_thumb_url char(80), aw_image_url char(255), delivery_time char(255), valid_from char(255), valid_to char(255), currency char(80), search_price datetime, store_price datetime, rrp_price char(255), display_price double, delivery_cost double, web_offer double, pre_order char(255), in_stock double, stock_quantity int(11), is_for_sale int(11), warranty int(11), condition1 int(11), product_type int(11), parent_product_id char(255), commission_group char(255), char(255), char(80), char(80), int(11), char(80) )

    I do that using a comma for the delimiter and then get a message “#1064 – You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ” at line 1″.
    Starting to think its more trouble than its worth.

  18. #18 by Sam at December 12th, 2009

    a transalation would be helpful thanks

    Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 1410065409 bytes) in /home/mensware/public_html/shop/upload.php on line 34

  19. #19 by Sam at December 12th, 2009

    I have now changed the permission of the file temporarily to 777 this now brings back 404 error

  20. #20 by Sam at December 12th, 2009

    you can just delete the past two comments, a bit of trial and error and i got it sorted

    but thanks for the code anyway

  21. #21 by admin at December 28th, 2009

    Sam, any chance of writing what you did, so that it may help others?

(will not be published)

Comment moderation is enabled. Your comment may take some time to appear.

  1. No trackbacks yet.