Electronics & Programming

develissimo

Open Source electronics development and programming

  • You are not logged in.
  • Root
  • » Django
  • » IntegrityError exception not caught by Django [RSS Feed]

#1 June 13, 2010 21:49:46

f.
Registered: 2009-11-02
Reputation: +  0  -
Profile   Send e-mail  

IntegrityError exception not caught by Django


In the process of serving a view I am saving a record that an incoming
URL specified a tracking ID.
One of the URLs got mangled by somebody somewhere resulting in it
being decoded as a non-existent user ID.

# I would expect this to throw an integrity error
tracking.save()


but it doesn't until TransactionMiddleware processes the outgoing
response.

At which point it throws an IntegrityError which Django does not
catch.

IntegrityError: insert or update on table "traffic_tracking2010"
violates foreign key constraint "traffic_tracking2010_user_id_fkey"
DETAIL: Key (user_id)=(1394967) is not present in table "auth_user".

Fortunately I have paste installed on top of Django so I got emailed
with the traceback (see below)

Can anybody figure out why the IntegrityError doesn't get thrown till
then ?
Why didn't Django catch errors thrown in middleware ?

I even tried this:

sid = transaction.savepoint()
try:
tracking.save()
except Exception,e:
get_logger().exception("tracking save: %s (%s)" % (src,e))
transaction.savepoint_rollback(sid)
else:
transaction.savepoint_commit(sid)

but again, no IntegrityError is thrown at that point.


Module paste.exceptions.errormiddleware:144 in __call__
<< __traceback_supplement__ = Supplement, self, environ
sr_checker = ResponseStartChecker(start_response)
app_iter = self.application(environ, sr_checker)
return self.make_catching_iter(app_iter, environ,
sr_checker)
except:>> app_iter = self.application(environ,
sr_checker)
Module django.core.handlers.wsgi:245 in __call__
<< # Apply response middleware
for middleware_method in
self._response_middleware:
response = middleware_method(request,
response)
response = self.apply_response_fixes(request,
response)
finally:>> response = middleware_method(request,
response)
Module django.middleware.transaction:25 in process_response
<< if transaction.is_managed():
if transaction.is_dirty():
transaction.commit()
transaction.leave_transaction_management()
return response>> transaction.commit()
Module django.db.transaction:199 in commit
<< using = DEFAULT_DB_ALIAS
connection = connections
connection._commit()
set_clean(using=using)
>> connection._commit()
Module django.db.backends:32 in _commit
<< def _commit(self):
if self.connection is not None:
return self.connection.commit()

def _rollback(self):>> return self.connection.commit()
IntegrityError: insert or update on table "traffic_tracking2010"
violates foreign key constraint "traffic_tracking2010_user_id_fkey"
DETAIL: Key (user_id)=(1394967) is not present in table "auth_user".

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to
django-users+unsubscr...@googlegroups.com.
For more options, visit this group athttp://groups.google.com/group/django-users?hl=en.

Offline

#2 June 13, 2010 22:30:08

a.
Registered: 2009-11-02
Reputation: +  0  -
Profile   Send e-mail  

IntegrityError exception not caught by Django


> Can anybody figure out why the IntegrityError doesn't get thrown till
> then ?
> Why didn't Django catch errors thrown in middleware ?

Most likely the foreign keys are defined as "deferrable initially
deferred", meaning that the database doesn't check them before commit.
And this of course means that Django can't catch the error before
commit.

>From the error message it seems like you are using Postgres, you can
check if this is the case by running manage.py dbshell and then type
"\d traffic_tracking2010". From the output you should see the
deferrable initially deferred on the foreign key. You could manually
disable the deferred behavior, but I don't know what this breaks.
Probably something, as there must be some reason for Django using
deferred checking.

--
Anssi

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to
django-users+unsubscr...@googlegroups.com.
For more options, visit this group athttp://groups.google.com/group/django-users?hl=en.

Offline

#3 June 13, 2010 23:24:13

f.
Registered: 2009-11-02
Reputation: +  0  -
Profile   Send e-mail  

IntegrityError exception not caught by Django


brilliant, you are correct:

Foreign-key constraints:
"traffic_tracking2010_content_type_id_fkey" FOREIGN KEY
(content_type_id) REFERENCES django_content_type(id) DEFERRABLE
INITIALLY DEFERRED
"traffic_tracking2010_src_content_type_id_fkey" FOREIGN KEY
(src_content_type_id) REFERENCES django_content_type(id) DEFERRABLE
INITIALLY DEFERRED
"traffic_tracking2010_user_id_fkey" FOREIGN KEY (user_id)
REFERENCES auth_user(id) DEFERRABLE INITIALLY DEFERRED

I wouldn't want to change that anyway.

Now what I still don't get is:

# doesn't protect against foreign key errors anyway ??
try:
sid = transaction.savepoint()
tracking.save()
transaction.savepoint_commit(sid)
except Exception,e:
get_logger().exception("tracking save: %s (%s)" % (src,e))
transaction.savepoint_rollback(sid)

why that attempt to commit just that save doesn't raise.

I really don't want to have to check for existence of User and
ContentType

thanks again.


as for why django didn't catch it:http://code.djangoproject.com/ticket/12909On Jun 13, 11:30 pm, akaariai <akaar...@gmail.com> wrote:
> > Can anybody figure out why the IntegrityError doesn't get thrown till
> > then ?
> > Why didn't Django catch errors thrown in middleware ?
>
> Most likely the foreign keys are defined as "deferrable initially
> deferred", meaning that the database doesn't check them before commit.
> And this of course means that Django can't catch the error before
> commit.
>
> From the error message it seems like you are using Postgres, you can
> check if this is the case by running manage.py dbshell and then type
> "\d traffic_tracking2010". From the output you should see the
> deferrable initially deferred on the foreign key. You could manually
> disable the deferred behavior, but I don't know what this breaks.
> Probably something, as there must be some reason for Django using
> deferred checking.
>
> --
> Anssi

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to
django-users+unsubscr...@googlegroups.com.
For more options, visit this group athttp://groups.google.com/group/django-users?hl=en.

Offline

#4 June 14, 2010 10:22:33

a.
Registered: 2009-11-02
Reputation: +  0  -
Profile   Send e-mail  

IntegrityError exception not caught by Django


On Jun 14, 1:24 am, felix <crucialfe...@gmail.com> wrote:
> brilliant, you are correct:
>
> Foreign-key constraints:
>     "traffic_tracking2010_content_type_id_fkey" FOREIGN KEY
> (content_type_id) REFERENCES django_content_type(id) DEFERRABLE
> INITIALLY DEFERRED
>     "traffic_tracking2010_src_content_type_id_fkey" FOREIGN KEY
> (src_content_type_id) REFERENCES django_content_type(id) DEFERRABLE
> INITIALLY DEFERRED
>     "traffic_tracking2010_user_id_fkey" FOREIGN KEY (user_id)
> REFERENCES auth_user(id) DEFERRABLE INITIALLY DEFERRED
>
> I wouldn't want to change that anyway.
>
> Now what I still don't get is:
>
>     # doesn't protect against foreign key errors anyway ??
>     try:
>         sid = transaction.savepoint()
>         tracking.save()
>         transaction.savepoint_commit(sid)
>     except Exception,e:
>         get_logger().exception("tracking save: %s (%s)" % (src,e))
>         transaction.savepoint_rollback(sid)
>
> why that attempt to commit just that save doesn't raise.
>

You are creating a savepoint. Savepoint isn't a full commit, so
deferrable constraints aren't checked.

> I really don't want to have to check for existence of User and
> ContentType

You could run with raw sql the following before trying to save:

from django.db import connection
cursor = connection.cursor()
tracking.save()
cursor.execute("SET CONSTRAINTS ALL IMMEDIATE")
cursor.execute("SET CONSTRAINTS ALL DEFERRED")

>From postgresql documentation (http://www.postgresql.org/docs/8.4/static/sql-set-constraints.html)

"When SET CONSTRAINTS changes the mode of a constraint from DEFERRED
to IMMEDIATE, the new mode takes effect retroactively: any outstanding
data modifications that would have been checked at the end of the
transaction are instead checked during the execution of the SET
CONSTRAINTS command. If any such constraint is violated, the SET
CONSTRAINTS fails (and does not change the constraint mode). Thus, SET
CONSTRAINTS can be used to force checking of constraints to occur at a
specific point in a transaction."

The risk here is that you have something else than the tracking.save()
with deferred foreign key check pending. Now, that check could fail at
the point you run SET CONSTRAINTS ALL IMMEDIATE. The risk should be
small, though. Note that I haven't tested this code...


Anssi

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to
django-users+unsubscr...@googlegroups.com.
For more options, visit this group athttp://groups.google.com/group/django-users?hl=en.

Offline

#5 June 14, 2010 19:43:50

f.
Registered: 2009-11-02
Reputation: +  0  -
Profile   Send e-mail  

IntegrityError exception not caught by Django


thanks again anssi, stellar reply

I might do this below. the tracking is the last thing to happen
before rendering,
so there are no more UPDATE or INSERTS
and in most cases views that use tracking don't do anything but select
anyway

but OTOH I'm already running rabbit and celery so I might message them
over and keep the tracking in a different process entirely.
I might be able to do it more efficiently there too.

-cfx


On Jun 14, 11:22 am, akaariai <akaar...@gmail.com> wrote:

>
> from django.db import connection
> cursor = connection.cursor()
> tracking.save()
> cursor.execute("SET CONSTRAINTS ALL IMMEDIATE")
> cursor.execute("SET CONSTRAINTS ALL DEFERRED")
>
> From postgresql documentation (http://www.postgresql.org/docs/8.4/> static/sql-set-constraints.html)
>
> "When SET CONSTRAINTS changes the mode of a constraint from DEFERRED
> to IMMEDIATE, the new mode takes effect retroactively: any outstanding
> data modifications that would have been checked at the end of the
> transaction are instead checked during the execution of the SET
> CONSTRAINTS command. If any such constraint is violated, the SET
> CONSTRAINTS fails (and does not change the constraint mode). Thus, SET
> CONSTRAINTS can be used to force checking of constraints to occur at a
> specific point in a transaction."
>
> The risk here is that you have something else than the tracking.save()
> with deferred foreign key check pending. Now, that check could fail at
> the point you run SET CONSTRAINTS ALL IMMEDIATE. The risk should be
> small, though. Note that I haven't tested this code...
>
> Anssi

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to
django-users+unsubscr...@googlegroups.com.
For more options, visit this group athttp://groups.google.com/group/django-users?hl=en.

Offline

  • Root
  • » Django
  • » IntegrityError exception not caught by Django [RSS Feed]

Board footer

Moderator control

Enjoy the 18th of October
PoweredBy

The Forums are managed by develissimo stuff members, if you find any issues or misplaced content please help us to fix it. Thank you! Tell us via Contact Options
Leave a Message
Welcome to Develissimo Live Support