#15 ✓invalid
Joe Van Dyk

Occasional problem with uploading small images to Amazon S3

Reported by Joe Van Dyk | September 6th, 2007 @ 02:53 PM

Here's the backtrace.

It doesn't happen all the time, just some of the time. Not sure how to reproduce it.

A EOFError occurred in crossword_words#create:

end of file reached

/infrastructure/root/lib/ruby/1.8/net/protocol.rb:133:in `sysread'

/infrastructure/root/lib/ruby/1.8/net/protocol.rb:133:in `sysread'

/infrastructure/root/lib/ruby/1.8/net/protocol.rb:133:in `rbuf_fill'

/infrastructure/root/lib/ruby/1.8/timeout.rb:56:in `timeout'

/infrastructure/root/lib/ruby/1.8/timeout.rb:76:in `timeout'

/infrastructure/root/lib/ruby/1.8/net/protocol.rb:132:in `rbuf_fill'

/infrastructure/root/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'

/infrastructure/root/lib/ruby/1.8/net/protocol.rb:126:in `readline'

/infrastructure/root/lib/ruby/1.8/net/http.rb:2017:in `read_status_line'

/infrastructure/root/lib/ruby/1.8/net/http.rb:2006:in `read_new'

/infrastructure/root/lib/ruby/1.8/net/http.rb:1047:in `request'

[RAILS_ROOT]/vendor/lib/aws/lib/aws/s3/connection.rb:43:in `request'

[RAILS_ROOT]/vendor/lib/aws/lib/aws/s3/connection.rb:48:in `call'

[RAILS_ROOT]/vendor/lib/aws/lib/aws/s3/connection.rb:48:in `request'

[RAILS_ROOT]/vendor/lib/aws/lib/aws/s3/base.rb:69:in `request'

[RAILS_ROOT]/vendor/lib/aws/lib/aws/s3/base.rb:83:in `put'

[RAILS_ROOT]/vendor/lib/aws/lib/aws/s3/object.rb:239:in `store_without_cache_control'

[RAILS_ROOT]/lib/s3_cache_control.rb:10:in `store'

[RAILS_ROOT]/vendor/plugins/attachment_fu/lib/technoweenie/attachment_fu/backends/s3_backend.rb:294:in `save_to_storage'

[RAILS_ROOT]/vendor/plugins/attachment_fu/lib/technoweenie/attachment_fu.rb:370:in `after_process_attachment'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/callbacks.rb:307:in `send'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/callbacks.rb:307:in `callback'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/callbacks.rb:304:in `each'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/callbacks.rb:304:in `callback'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/callbacks.rb:214:in `create_or_update'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/base.rb:1711:in `save_without_validation!'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/validations.rb:877:in `save_without_transactions!'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/transactions.rb:109:in `save!'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:59:in `transaction'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/query_cache.rb:73:in `send'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/query_cache.rb:73:in `method_missing'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/transactions.rb:77:in `transaction'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/transactions.rb:97:in `transaction'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/transactions.rb:109:in `save!'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/transactions.rb:117:in `rollback_active_record_state!'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/transactions.rb:109:in `save!'

[RAILS_ROOT]/vendor/plugins/attachment_fu/lib/technoweenie/attachment_fu.rb:215:in `create_or_update_thumbnail'

[RAILS_ROOT]/vendor/lib/aws/lib/aws/s3/extensions.rb:133:in `returning'

[RAILS_ROOT]/vendor/plugins/attachment_fu/lib/technoweenie/attachment_fu.rb:207:in `create_or_update_thumbnail'

[RAILS_ROOT]/vendor/plugins/attachment_fu/lib/technoweenie/attachment_fu.rb:368:in `after_process_attachment'

[RAILS_ROOT]/vendor/plugins/attachment_fu/lib/technoweenie/attachment_fu.rb:368:in `each'

[RAILS_ROOT]/vendor/plugins/attachment_fu/lib/technoweenie/attachment_fu.rb:368:in `after_process_attachment'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/callbacks.rb:307:in `send'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/callbacks.rb:307:in `callback'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/callbacks.rb:304:in `each'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/callbacks.rb:304:in `callback'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/callbacks.rb:214:in `create_or_update'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/base.rb:1705:in `save_without_validation'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/validations.rb:867:in `save_without_transactions'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/transactions.rb:105:in `save'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:59:in `transaction'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/query_cache.rb:73:in `send'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/query_cache.rb:73:in `method_missing'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/transactions.rb:77:in `transaction'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/transactions.rb:97:in `transaction'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/transactions.rb:105:in `save'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/transactions.rb:117:in `rollback_active_record_state!'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/transactions.rb:105:in `save'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/associations/association_proxy.rb:124:in `send'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/associations/association_proxy.rb:124:in `method_missing'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/callbacks.rb:312:in `callback'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/callbacks.rb:309:in `callback'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/callbacks.rb:304:in `each'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/callbacks.rb:304:in `callback'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/callbacks.rb:214:in `create_or_update'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/base.rb:1705:in `save_without_validation'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/validations.rb:867:in `save_without_transactions'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/transactions.rb:105:in `save'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:59:in `transaction'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/query_cache.rb:73:in `send'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/query_cache.rb:73:in `method_missing'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/transactions.rb:77:in `transaction'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/transactions.rb:97:in `transaction'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/transactions.rb:105:in `save'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/transactions.rb:117:in `rollback_active_record_state!'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/transactions.rb:105:in `save'

[RAILS_ROOT]/app/controllers/crossword_words_controller.rb:29:in `create'

[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/base.rb:1080:in `send'

[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/base.rb:1080:in `perform_action_without_filters'

[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/filters.rb:695:in `call_filters'

[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/filters.rb:721:in `run_before_filters'

[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/filters.rb:662:in `call'

[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/filters.rb:662:in `proxy_before_and_after_filter'

[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/filters.rb:483:in `call'

[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/filters.rb:483:in `call'

[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/filters.rb:718:in `run_before_filters'

[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/filters.rb:693:in `call_filters'

[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/filters.rb:687:in `perform_action_without_benchmark'

[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/benchmarking.rb:68:in `perform_action_without_rescue'

/infrastructure/root/lib/ruby/1.8/benchmark.rb:293:in `measure'

[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/benchmarking.rb:68:in `perform_action_without_rescue'

[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/rescue.rb:133:in `perform_action_without_caching'

[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/caching.rb:668:in `perform_action'

[RAILS_ROOT]/vendor/rails/activerecord/lib/active_record/query_cache.rb:106:in `cache'

[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/caching.rb:667:in `perform_action'

[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/base.rb:496:in `send'

[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/base.rb:496:in `process_without_filters'

[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/filters.rb:683:in `process_without_session_management_support'

[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/session_management.rb:122:in `process'

[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/base.rb:348:in `process'

[RAILS_ROOT]/vendor/rails/railties/lib/dispatcher.rb:39:in `dispatch'

/infrastructure/root/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel/rails.rb:78:in `process'

/infrastructure/root/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel/rails.rb:76:in `synchronize'

/infrastructure/root/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel/rails.rb:76:in `process'

/infrastructure/root/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel.rb:618:in `process_client'

/infrastructure/root/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel.rb:617:in `each'

/infrastructure/root/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel.rb:617:in `process_client'

/infrastructure/root/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel.rb:736:in `run'

/infrastructure/root/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel.rb:736:in `initialize'

/infrastructure/root/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel.rb:736:in `new'

/infrastructure/root/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel.rb:736:in `run'

/infrastructure/root/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel.rb:720:in `initialize'

/infrastructure/root/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel.rb:720:in `new'

/infrastructure/root/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel.rb:720:in `run'

/infrastructure/root/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel/configurator.rb:271:in `run'

/infrastructure/root/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel/configurator.rb:270:in `each'

/infrastructure/root/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel/configurator.rb:270:in `run'

/infrastructure/root/lib/ruby/gems/1.8/gems/mongrel-1.0.1/bin/mongrel_rails:127:in `run'

/infrastructure/root/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel/command.rb:211:in `run'

/infrastructure/root/lib/ruby/gems/1.8/gems/mongrel-1.0.1/bin/mongrel_rails:243

/infrastructure/root/bin/mongrel_rails:16:in `load'

/infrastructure/root/bin/mongrel_rails:16

Comments and changes to this ticket

  • akira

    akira September 21st, 2007 @ 09:08 AM

    I had the same issue, it actually messed up my Mongrel where I had to restart it after this error, otherwise it would run out of memory. The issue occurred for me after Mongrel is up and running for a while (try attaching something, waiting for a while, and then attaching it again - it doesn't seem to occur right away). Anyways, I turned the persistent connection off, and it seemed a little better. I think I still got an error once in a while, not sure if it was network related (slightly different stack trace), but it did not bring the server down as before.

    I modified the s3_backend.rb class locally and added a persistent connection option, which I set to false on all my configurations:

    Base.establish_connection!(

    :access_key_id => s3_config[:access_key_id],

    :secret_access_key => s3_config[:secret_access_key],

    :server => s3_config[:server],

    :port => s3_config[:port],

    :use_ssl => s3_config[:use_ssl],

    :persistent => s3_config[:persistent]

    )

  • Rick

    Rick October 8th, 2007 @ 12:40 PM

    • State changed from “new” to “invalid”

    I moved the S3 upload process to an external rake task and haven't had issues. I wouldn't really recommend uploading stuff to S3 in a Rails action anyway, you'll hold the process up.

  • Peter

    Peter November 6th, 2007 @ 05:23 PM

    Hi Rick,

    I'm experiencing a similar problem when uploading images (not using RMagick) to Amazon S3 intermittently, killing my mongrel requiring a restart.

    Error in mongrel.log

    /vendor/rails/activerecord/lib/active_record/transactions.rb:85:$ from /var/lib/gems/1.8/gems/mongrel-1.0.4/bin/../lib/mongrel/configurator.rb:303:in `call'

    from /var/lib/gems/1.8/gems/mongrel-1.0.4/bin/../lib/mongrel/configurator.rb:303:in `join'

    from /var/lib/gems/1.8/gems/mongrel-1.0.4/bin/../lib/mongrel/configurator.rb:303:in `join'

    from /var/lib/gems/1.8/gems/mongrel-1.0.4/bin/../lib/mongrel/configurator.rb:303:in `each'

    from /var/lib/gems/1.8/gems/mongrel-1.0.4/bin/../lib/mongrel/configurator.rb:303:in `join'

    from /var/lib/gems/1.8/gems/mongrel-1.0.4/bin/mongrel_rails:135:in `run'

    from /var/lib/gems/1.8/gems/mongrel-1.0.4/bin/../lib/mongrel/command.rb:211:in `run'

    from /var/lib/gems/1.8/gems/mongrel-1.0.4/bin/mongrel_rails:279

    from /var/lib/gems/1.8/bin/mongrel_rails:18:in `load'

    from /var/lib/gems/1.8/bin/mongrel_rails:18

    Any ideas?

    Thanks,

    Peter.

  • Paul Dowman

    Paul Dowman February 22nd, 2008 @ 02:04 PM

    I'm not sure why this is marked as invalid, it's definitely a real problem that MANY people are seeing as a quick google search shows.

    I had this exact problem and the cause of it was that the s3 connection was being established with :persistent => true

  • Paul Dowman

    Paul Dowman February 22nd, 2008 @ 02:09 PM

    There was supposed to be a patch attached to the last post, but I don't see it. Here it is inline: 

    
Index: lib/technoweenie/attachment_fu/backends/s3_backend.rb

    
===================================================================

    
--- lib/technoweenie/attachment_fu/backends/s3_backend.rb	(revision 3145)

    
+++ lib/technoweenie/attachment_fu/backends/s3_backend.rb	(working copy)

    
@@ -145,7 +145,8 @@

    
:secret_access_key => s3_config[:secret_access_key],

    
:server            => s3_config[:server],

    
:port              => s3_config[:port],

    

    • 
           :use_ssl           => s3_config[:use_ssl]

    • 
           :use_ssl           => s3_config[:use_ssl],

    • 
           :persistent        => false

    

    
)

    

    1. 
Bucket.create(@@bucket_name)

  • Paul Dowman

    Paul Dowman February 22nd, 2008 @ 02:14 PM

    Well the formatting didn't work so well that time, I give up. Anyway the point is to add ":persistent => false".

  • Hans

    Hans March 16th, 2008 @ 08:53 AM

    This is absolutely not an invalid ticked. I am finding I have to restart my production cluster every 90 minutes or so if there is active uploading to S3 with attachment_fu.

    During the months of November/December of 2008, when we weren't adding any new products to our system (and, quid-pro-quo, not uploading to S3), the total number of production cluster restarts were three - and each of those was due to a functionality change.

  • Paul Dowman

    Paul Dowman March 16th, 2008 @ 09:48 AM

    @Hans: does the :persistent => false change help?

  • Hans

    Hans March 16th, 2008 @ 11:34 AM

    I am rolling that in to the production cluster today after finishing testing. We'll see, I won't know until the users are back at work late in the day on Monday.

  • Paul Dowman

    Paul Dowman April 21st, 2008 @ 01:19 PM

    Can somebody please let me know why this is marked as invalid? Anybody...?

  • Hans

    Hans April 21st, 2008 @ 04:46 PM

    The persistent flag appears to have solved the issue. We have been running with it for about a month (and probably 3,000 uploads) without any hangs.

    Thanks!

Please Sign in or create a free account to add a new ticket.

With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.

New-ticket Create new ticket

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile ยป

People watching this ticket

Attachments

Pages