Help - remove duplicates accross list of list

Lists-Logic
1

Hi, I’m trying to remove duplicates accross a list of lists, keeping the order.
I have this :

image
image.png780×593 142 KB

For instance, here the item 2504994 and 2505056 are already in list 1. I want to remove from them from list 2.
I have the following error:
Traceback (most recent call last):
File “”, line 8, in
IndexError: index out of range: 9

Python not really my strongsuit, can anyone help please ?

2

I assume it’s because after you’ve remove a duplicate your range is no longer valid. Instead of trying to remove items I would suggest building a new list.

Iterate through each item in each sublist. Check to see if the item is already in a list of duplicates. If it’s already in your list of duplicates, skip it. If it’s not, append it to your new list and add it to the list of duplicates.

3

Hi, @tcosse9401!
It is hard to understand through all this levels of looping

You can try ‘set’ methode:

a=IN[0]
list_final=a
count=1
for i in range(len(list_final)-1):
    list_final[count]=list(set(list_final[0+count])-set(list_final[0]))
    count+=1
OUT=list_final
1 Like
4

Hi @Dmytro_Serebriian,

Your script works perfectly well for 2 list but what if I want to use it for more than 2 lists.
i.e. I don’t want 6 and 9 to be in list 2 since they have already been in list 1
Capture

5

Thanks for respondiing so quickly, this is great advice Nick, I’m going to try it out ASAP.

6

@Dmytro_Serebriian, I haven’t quite understood you code yet, but @AmolShah is right, I have the same problem when running the script.
Anyway, thank you so much to all of you for responding so quickly ! Your help is very constructive, I’m going to try out Nick’s method and I’ll keep you posted !

7

There is an OOTB solution around these nodes too:
List.Flatten
List.UniqueItems
List.AllIndiciesOf with cross product lacing
List.DropItems
List.Transpose
List.RemoveItemAtIndex

8

Hi, @AmolShah
Yes, then its a quite different situation. In this case you need to add another level of looping.
It looks a bit fuzzy, so i think it is better try to use nodes.
In any way here is the python solution:
image

list_final=IN[0]
count=1
count2=0
num=0
ran=1
for j in range(len(list_final)-1):
	for i in range(len(list_final)-ran):
		list_final[count+num]=list(set(list_final[count+num])-set(list_final[count2]))
		count+=1
	count=1
	count2+=1
	num+=1
	ran+=1
OUT=list_final
1 Like
9

Look in Orchid package, it has nodes for this that works at any list depth

10

image

sublists = IN[0]
out = []
duplicates = []
for sublist in sublists:
	unique = []
	for item in sublist:
		if item not in duplicates:
			unique.append(item)
			duplicates.append(item)
	out.append(unique)
OUT = out
5 Likes
11

Thank you @Dmytro_Serebriian and @Nick_Boyts for the Py script.
That’s what I was looking for.

12

Thanks you @Nick_Boyts and @Dmytro_Serebriian for your awesome help. Both of your scripts solve my problem. The upside of Nick’s method is that it keeps the order in the sublists, whereas Dmytro’s doesn’t. I’ll be sorting the lists afterwards anyway, so both work for me, but it’s good to keep that in mind. A big thank you :smile: