After some tinkering, I have managed to remove the filter. The remove button just wasn't working in Safari (on mac). After installing Chrome and trying it there, the remove button finally worked, so I was able to remove the line.
So now that the filter is removed, what is next? I have an empty user list still I'm afraid.