Electronics & Programming

develissimo

Open Source electronics development and programming

  • You are not logged in.
  • Root
  • » Django
  • » How to do background processing with Django [RSS Feed]

#1 March 26, 2008 09:58:12

s.
Registered: 2009-11-02
Reputation: +  0  -
Profile   Send e-mail  

How to do background processing with Django


I response to some urls, my views need to start some potentially time
taking actions. So how can I start background process, would it be as
simple as

...
t = threading.Thread(...)
t.setDaemon(True)
t.start()
return HttpResponse(..)

Or does django/apache have limitation on how threading can be used?
Would my thread be killed, once the response is sent back?

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to
For more options, visit this group athttp://groups.google.com/group/django-users?hl=en-~----------~----~----~----~------~----~------~--~---

Offline

#2 March 26, 2008 10:24:14

Tane P.
Registered: 2009-11-02
Reputation: +  0  -
Profile   Send e-mail  

How to do background processing with Django


Hi there,

It might not be exactly what you need, but there is Django Queue
Service (http://code.google.com/p/django-queue-service/)

I've implemented it in our django application to provide a queue
service to do backend processes. Our app is a mercurial manager, and
I built a queue to handle our remote cloning requests. There is a
queue called 'repoclone' and when a user creates on, this is the
workflow:

1. User creates request to clone remote repository
2. Django checks details are valid
3. Creates a message in the 'repoclone' queue
4. Django saves the instance of the repo details in the database with
created = False
5. Cron runs every 1 min, poping the next message off the queue
6. Django reads the queue JSON and decodes, grabs the repo information
from the database
7. Does the clone request
8. Updates the database to advise the clone has been created.

Hope that helps.

On Wed, Mar 26, 2008 at 8:58 AM, shabda <> wrote:
>
> I response to some urls, my views need to start some potentially time
> taking actions. So how can I start background process, would it be as
> simple as
>
> ...
> t = threading.Thread(...)
> t.setDaemon(True)
> t.start()
> return HttpResponse(..)
>
> Or does django/apache have limitation on how threading can be used?
> Would my thread be killed, once the response is sent back?
>
> >
>



--
Tane Piper
Blog -http://digitalspaghetti.me.ukSkype: digitalspaghetti
Wii: 4734 3486 7149 1830

This email is: blogable ask first private

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to
For more options, visit this group athttp://groups.google.com/group/django-users?hl=en-~----------~----~----~----~------~----~------~--~---

Offline

#3 March 26, 2008 10:28:40

Graham D.
Registered: 2009-11-02
Reputation: +  0  -
Profile   Send e-mail  

How to do background processing with Django


On Mar 26, 7:58 pm, shabda <> wrote:
> I response to some urls, my views need to start some potentially time
> taking actions. So how can I start background process, would it be as
> simple as
>
> ...
> t = threading.Thread(...)
> t.setDaemon(True)
> t.start()
> return HttpResponse(..)
>
> Or does django/apache have limitation on how threading can be used?
> Would my thread be killed, once the response is sent back?

You are much much better off having a back end process which embeds a
mini HTTP server set up to respond to XML-RPC requests. The Apache/
Django processes can then make an XML-RPC request to the long running
persistent back end process which then does the work in a separate
thread, via some form of queuing mechanism and thread pooling
mechanism as needs be to keep number of active threads down, or using
other means as appropriate.

Being in a single back end process, you can then also support XML-RPC
requests to check on status of jobs being done, cancel jobs etc etc.
You also don't then have to deal with the fact that Apache is
multiprocess and subsequent requests may not come back to the same
process if something needs to query outcome of job. You also don't
have to deal with possibility of Apache killing off one its processes,
which might contain running jobs, when it is recycling its processes.

Graham
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to
For more options, visit this group athttp://groups.google.com/group/django-users?hl=en-~----------~----~----~----~------~----~------~--~---

Offline

#4 March 26, 2008 14:20:45

s.
Registered: 2009-11-02
Reputation: +  0  -
Profile   Send e-mail  

How to do background processing with Django


The problem with these aproaches is that they add an extra layer of
things to do. I want to package it so that the only required things
are django and apache. Adding a HTTP server or configuring django-
queuing is too painful for the average user. :(

On Mar 26, 2:28 pm, Graham Dumpleton <>
wrote:
> On Mar 26, 7:58 pm, shabda <> wrote:
>
> > I response to some urls, my views need to start some potentially time
> > taking actions. So how can I start background process, would it be as
> > simple as
>
> > ...
> > t = threading.Thread(...)
> > t.setDaemon(True)
> > t.start()
> > return HttpResponse(..)
>
> > Or does django/apache have limitation on how threading can be used?
> > Would my thread be killed, once the response is sent back?
>
> You are much much better off having a back end process which embeds a
> mini HTTP server set up to respond to XML-RPC requests. The Apache/
> Django processes can then make an XML-RPC request to the long running
> persistent back end process which then does the work in a separate
> thread, via some form of queuing mechanism and thread pooling
> mechanism as needs be to keep number of active threads down, or using
> other means as appropriate.
>
> Being in a single back end process, you can then also support XML-RPC
> requests to check on status of jobs being done, cancel jobs etc etc.
> You also don't then have to deal with the fact that Apache is
> multiprocess and subsequent requests may not come back to the same
> process if something needs to query outcome of job. You also don't
> have to deal with possibility of Apache killing off one its processes,
> which might contain running jobs, when it is recycling its processes.
>
> Graham
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to
For more options, visit this group athttp://groups.google.com/group/django-users?hl=en-~----------~----~----~----~------~----~------~--~---

Offline

#5 March 26, 2008 15:39:25

Ian L.
Registered: 2009-11-02
Reputation: +  0  -
Profile   Send e-mail  

How to do background processing with Django


Hi
> are django and apache. Adding a HTTP server or configuring django-
> queuing is too painful for the average user. :(

what does the end user have to do with this?
you have a server by doing
wgethttp://svn.cherrypy.org/trunk/cherrypy/wsgiserver/__init__.py-O
wsgiserver.py
and setting up xml-rpc is not much more than this...i did this recently athttps://help.ubuntu.com/community/UMEGuide/ApplicationDevelopment/GPSEnabledWebApplicationand it is pretty simple to do (and really the best way IMO to run
background processes)

Ian

--http://ianlawrence.info--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to
For more options, visit this group athttp://groups.google.com/group/django-users?hl=en-~----------~----~----~----~------~----~------~--~---

Offline

#6 March 26, 2008 17:08:12

s.
Registered: 2009-11-02
Reputation: +  0  -
Profile   Send e-mail  

How to do background processing with Django


Not the end user but the people who would be downloading this
application, and want to use this, in a shared hosting environment.

On Mar 26, 7:39 pm, "Ian Lawrence" <> wrote:
> Hi
>
> > are django and apache. Adding a HTTP server or configuring django-
> > queuing is too painful for the average user. :(
>
> what does the end user have to do with this?
> you have a server by doing
> wgethttp://svn.cherrypy.org/trunk/cherrypy/wsgiserver/__init__.py-O> wsgiserver.py
> and setting up xml-rpc is not much more than this...i did this recently
> athttps://help.ubuntu.com/community/UMEGuide/ApplicationDevelopment/GPS...
> and it is pretty simple to do (and really the best way IMO to run
> background processes)
>
> Ian
>
> --http://ianlawrence.info--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to
For more options, visit this group athttp://groups.google.com/group/django-users?hl=en-~----------~----~----~----~------~----~------~--~---

Offline

#7 March 26, 2008 18:07:18

Joseph H.
Registered: 2009-11-02
Reputation: +  0  -
Profile   Send e-mail  

How to do background processing with Django


You can certainly hide this additional complexity with your
application either using Django as a pass through or configuring up
something with a reverse proxy in Apache or such. There's no reason
that I can think of that the complexity of maintaining a queueing
system for asynchronous processing would need to be passed outward to
a user.

If you want to give the user feedback on what's happening, you might
consider maintaining a state message like "in progress" or such within
Django that can be polled via an Ajax call or the like.

-joe

On Wed, Mar 26, 2008 at 9:07 AM, shabda <> wrote:
>
> Not the end user but the people who would be downloading this
> application, and want to use this, in a shared hosting environment.
>
>
> On Mar 26, 7:39 pm, "Ian Lawrence" <> wrote:
> > Hi
> >
> > > are django and apache. Adding a HTTP server or configuring django-
> > > queuing is too painful for the average user. :(
> >
> > what does the end user have to do with this?
> > you have a server by doing
> > wgethttp://svn.cherrypy.org/trunk/cherrypy/wsgiserver/__init__.py-O> > wsgiserver.py
> > and setting up xml-rpc is not much more than this...i did this recently
> athttps://help.ubuntu.com/community/UMEGuide/ApplicationDevelopment/GPS...
>
>
> > and it is pretty simple to do (and really the best way IMO to run
> > background processes)
> >
> > Ian
> >
> > --http://ianlawrence.info> >
>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to
For more options, visit this group athttp://groups.google.com/group/django-users?hl=en-~----------~----~----~----~------~----~------~--~---

Offline

  • Root
  • » Django
  • » How to do background processing with Django [RSS Feed]

Board footer

Moderator control

Enjoy the 16th of December
PoweredBy

The Forums are managed by develissimo stuff members, if you find any issues or misplaced content please help us to fix it. Thank you! Tell us via Contact Options
Leave a Message
Welcome to Develissimo Live Support