Electronics & Programming

develissimo

Open Source electronics development and programming

  • You are not logged in.

#1 June 18, 2010 02:28:45

Kenneth G.
Registered: 2009-11-02
Reputation: +  0  -
Profile   Send e-mail  

how to eliminate duplicates


hi

I have a problem - there is a model called Player with first_name and
last_name. There is a unique_together constraint on first_name and last_name.
However I find that the people doing data entry have been entering things like
Ram Sharan and RAM SHARAN which are two different names. Of course I can
prevent future mistakes by adding validation, but now I am faced with the
problem of merging these records - any suggestions on how to do this? Flow
would be:
1. select duplicate names
2. shift all records of the duplicate to the original
3. delete the duplicate.
--
Regards
Kenneth Gonsalves
Senior Associate
NRC-FOSS at AU-KBC

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to
django-users+unsubscr...@googlegroups.com.
For more options, visit this group athttp://groups.google.com/group/django-users?hl=en.

Offline

#2 June 18, 2010 02:57:15

Shawn M.
Registered: 2009-11-02
Reputation: +  0  -
Profile   Send e-mail  

how to eliminate duplicates


Use South data migrations.

Sent from my iPad, in accordance with the prophesy.

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to
django-users+unsubscr...@googlegroups.com.
For more options, visit this group athttp://groups.google.com/group/django-users?hl=en.

Offline

#3 June 18, 2010 09:02:08

Kenneth G.
Registered: 2009-11-02
Reputation: +  0  -
Profile   Send e-mail  

how to eliminate duplicates


On Friday 18 June 2010 07:27:10 Shawn Milochik wrote:
> Use South data migrations.
>

I do not understand what data migration has to do with this.
--
Regards
Kenneth Gonsalves
Senior Associate
NRC-FOSS at AU-KBC

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to
django-users+unsubscr...@googlegroups.com.
For more options, visit this group athttp://groups.google.com/group/django-users?hl=en.

Offline

#4 June 18, 2010 09:38:03

Tom E.
Registered: 2009-11-02
Reputation: +  0  -
Profile   Send e-mail  

how to eliminate duplicates


On Fri, Jun 18, 2010 at 9:03 AM, Kenneth Gonsalves <law...@au-kbc.org> wrote:
> On Friday 18 June 2010 07:27:10 Shawn Milochik wrote:
>> Use South data migrations.
>>
>
> I do not understand what data migration has to do with this.

Data fixups are one sort of migration - its not a schema migration,
but a data migration.

It all depends what sort of data is on your models. If its complex,
you may have to implement a mechanical turk style application to allow
a human to choose which bits of which objects to save.

Cheers

Tom

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to
django-users+unsubscr...@googlegroups.com.
For more options, visit this group athttp://groups.google.com/group/django-users?hl=en.

Offline

#5 June 18, 2010 09:43:29

Matt H.
Registered: 2009-11-02
Reputation: +  0  -
Profile   Send e-mail  

how to eliminate duplicates


I'm assuming there's no fields on the duplicate player instances that
need merging - from your mention of "shift all records of the
duplicate" I'm reading that as "shift all records that relate to the
player" - otherwise it will need human intervention. Having stated
that assumption, while it's possible someone may have a re-usable bit
of code to do the repointing of relationship fields (as I figure it's
probably possible to knock something up that looks at the relationship
fields/reverse relationships on a given class and do the updates), for
a one off like this (and assuming a SQL back-end) if I were faced with
that problem I personally would write a script to figure out what
Player record IDs are duplicates (iterative over values_list of the
id, first_name and last_name, build up map of full name in a canonical
form to IDs, pick the lowest number
from the list for a given name as being the master and treat the rest
as the duplicates) and then generate and run the raw sql to update any
column in tables that points to the IDs (i.e. UPDATE foo SET blah_id=
%s WHERE blah_id IN (3,4,123,4) for foreign key fields and also for
m2m tables) and then do the deletions with .delete() calls on the
objects.

On Jun 18, 2:29 am, Kenneth Gonsalves <law...@au-kbc.org> wrote:
> hi
>
> I have a problem - there is a model called Player with first_name and
> last_name. There is a unique_together constraint on first_name and last_name.
> However I find that the people doing data entry have been entering things like
> Ram Sharan and RAM SHARAN which are two different names. Of course I can
> prevent future mistakes by adding validation, but now I am faced with the
> problem of merging these records - any suggestions on how to do this? Flow
> would be:
> 1. select duplicate names
> 2. shift all records of the duplicate to the original
> 3. delete the duplicate.
> --
> Regards
> Kenneth Gonsalves
> Senior Associate
> NRC-FOSS at AU-KBC

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to
django-users+unsubscr...@googlegroups.com.
For more options, visit this group athttp://groups.google.com/group/django-users?hl=en.

Offline

#6 June 18, 2010 09:46:59

Kenneth G.
Registered: 2009-11-02
Reputation: +  0  -
Profile   Send e-mail  

how to eliminate duplicates


On Friday 18 June 2010 14:07:53 Tom Evans wrote:
> On Fri, Jun 18, 2010 at 9:03 AM, Kenneth Gonsalves <law...@au-kbc.org>
wrote:
> > On Friday 18 June 2010 07:27:10 Shawn Milochik wrote:
> >> Use South data migrations.
> >
> > I do not understand what data migration has to do with this.
>
> Data fixups are one sort of migration - its not a schema migration,
> but a data migration.
>
> It all depends what sort of data is on your models. If its complex,
> you may have to implement a mechanical turk style application to allow
> a human to choose which bits of which objects to save.
>

ok, I was unaware of the fact that south does data migration also - I thought
it was just schema migration. I will look into it.
--
Regards
Kenneth Gonsalves
Senior Associate
NRC-FOSS at AU-KBC

--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to
django-users+unsubscr...@googlegroups.com.
For more options, visit this group athttp://groups.google.com/group/django-users?hl=en.

Offline

Board footer

Moderator control

Enjoy the 13th of December
PoweredBy

The Forums are managed by develissimo stuff members, if you find any issues or misplaced content please help us to fix it. Thank you! Tell us via Contact Options
Leave a Message
Welcome to Develissimo Live Support