mardi 4 août 2015

Parse CSV File Using Delayed Job

My Question is Can i incorporate some way of uploading this file then doing the rest of this as a background JOB

From the import function thru the end. so that my server will not time out.


I keep getting timeout errors when i upload a file to be inserted into the database.

My form uploads the file and passes the file into my import function here:

    def import(file)
      parse_file(file)
    end

I have an delayed job on the database creation of the data as you can see below.

    def parse_file(file)
      env = Rails.env.development?
      error = 0
      line = 2
      posts = parse_csv(file)
      posts.each do |post|
        record = build_records(post, line)
        #@log.info record
        unless record
          @log.debug "Error on line no #{line} Data: #{post}"
          line = line+1
          error = error+1
          next
        end
        line = line+1
        if env #checks for development
          #development
          RejectLoads.create(record)
        else
          #production
          RejectLoads.delay(:queue => 'rejects').create(record)
        end
      end
      @log.info "Total Errors: #{error}"
    end

parsing here:

def parse_csv(file)
  table = CSV.read(file.path, { :headers => true, :col_sep => "\t", :skip_blanks => true, :encoding => 'ISO-8859-1'})
  unless table.empty?
    header_arry = Array.new
    table.headers.each do |h|
    # simplest case here
    header_arry << h.downcase
    #which produces an array of column names called header_arry
  end
  rows = table.to_a
  rows.delete_at(0)
  posts = []
  rows.each do |row|
    #convert to hash using the column names
    post = Hash[header_arry.zip(row)]
    # do something with the row hash
    posts.push post
  end
  posts
  end
end

Aucun commentaire:

Enregistrer un commentaire