On one my sites I'm using a modified version of Lierduh's NNTP Forum Feed hack. The hack works nicely (after a few tweaks) but there is still one unresolved issue: If the hack timesout while importing new posts (or "articles" in UseNet speak) then it will re-import them again during the next run. 
OK, no problem, right? vBulletin already has the feature to delete duplicates threads, right? Ah... yes & no.
vBulletin does indeed have a stock option to delete duplicate threads (ACP => Maintenance => Update Counters => Delete Duplicate Threads) but the key phrase is "threads" whereas my problem was with duplicate posts as in replies to the thread. No option for that one. The solution was to create a new one based upon the Delete Duplicate Threads option.
The new functionality requires two edits to your {forums}/admincp/misc.php file. This edit is based upon the vB 3.6.x code so if you're planning on using it with vB 3.5.x I highly recommend you test first in a test environment before running it on your production site.
Step #1: Creating The New Menu Option
In your {forums}/admincp/misc.php file find...
... and below it add:
Step #2: Adding the New Function
In your {forums}/admincp/misc.php file find...
... and above it add:
That's it, you're all done. The next time you go into your ACP you should now see a new option below the Remove Duplicate Threads option titled Remove Duplicate Posts. If you don't see the option then you did something wrong in Step #1 above. If you see the option but it doesn't do anything then you did something wrong in Step #2 above.
One final important note: This is only of benefit to people who, for one reason or another, may end up with duplicate posts in their vBulletin tables. The NNTP hack is one example of it might occur. If you do not need this functionality then I would recommend that you do not install this hack then! There is absolutely no point in adding a hack to your vBulletin community that has the potential of removing posts from your threads unless you are absolutely 100% positive that you need it.
OK, no problem, right? vBulletin already has the feature to delete duplicates threads, right? Ah... yes & no.
The new functionality requires two edits to your {forums}/admincp/misc.php file. This edit is based upon the vB 3.6.x code so if you're planning on using it with vB 3.5.x I highly recommend you test first in a test environment before running it on your production site.
Step #1: Creating The New Menu Option
In your {forums}/admincp/misc.php file find...
Code:
print_form_header('misc', 'removedupe');
print_table_header($vbphrase['delete_duplicate_threads'], 2, 0);
print_description_row($vbphrase['note_duplicate_threads_have_same']);
print_input_row($vbphrase['number_of_threads_to_process_per_cycle'], 'perpage', 500);
print_submit_row($vbphrase['delete_duplicate_threads']);
Code:
// Delete Duplicate Posts Hack by vBulletinUsers.com
print_form_header('misc', 'removedupeposts');
print_table_header('Delete Duplicate Posts', 2, 0);
print_description_row('<strong>Note:</strong> Duplicate posts are posts that have the same title, post user name name, and post time within the same thread.');
print_input_row('Number of threads to process per cycle', 'perpage', 500);
print_submit_row('Delete Duplicate Posts');
// /Delete Duplicate Posts Hack by vBulletinUsers.com
In your {forums}/admincp/misc.php file find...
Code:
// ###################### Start find lost users #######################
if ($_POST['do'] == 'lostusers')
Code:
// Delete Duplicate Posts Hack by vBulletinUsers.com
// ###################### Start remove dupe posts #######################
if ($_REQUEST['do'] == 'removedupeposts')
{
if (empty($vbulletin->GPC['perpage']))
{
$vbulletin->GPC['perpage'] = 500;
}
$finishat = $vbulletin->GPC['startat'] + $vbulletin->GPC['perpage'];
echo '<p>Removing duplicate posts</p>';
$posts = $db->query_read("
SELECT postid, post.title, post.threadid, post.username, post.dateline, thread.title as threadtitle
FROM " . TABLE_PREFIX . "post as post
INNER JOIN " . TABLE_PREFIX . "thread AS thread ON(thread.threadid = post.threadid)
WHERE postid >= " . $vbulletin->GPC['startat'] . " AND
postid < $finishat
ORDER BY postid
");
while ($post = $db->fetch_array($posts))
{
$deleteposts = $db->query_read("
SELECT *
FROM " . TABLE_PREFIX . "post
WHERE title = '" . $db->escape_string($post['title']) . "' AND
threadid = $post[threadid] AND
username = '" . $db->escape_string($post['username']) . "' AND
dateline = $post[dateline] AND
postid > $post[postid]
");
while ($deletepost = $db->fetch_array($deleteposts))
{
$postman =& datamanager_init('Post', $vbulletin, ERRTYPE_SILENT, 'threadpost');
$postman->set_existing($post);
$postman->delete();
unset($postman);
echo " ".construct_phrase($vbphrase['delete_x'], $deletepost['postid'] . ' ' . $deletepost['title'])."<br />";
}
echo construct_phrase($vbphrase['processing_x'], $post['postid'] . ' ' . iif(!empty($post['title']),$post['title'],$post['threadtitle']) ) . "<br />\n";
vbflush();
}
if ($checkmore = $db->query_first("SELECT postid FROM " . TABLE_PREFIX . "post WHERE postid >= $finishat LIMIT 1"))
{
print_cp_redirect("misc.php?" . $vbulletin->session->vars['sessionurl'] . "do=removedupeposts&startat=$finishat&pp=" . $vbulletin->GPC['perpage']);
echo "<p><a href=\"misc.php?" . $vbulletin->session->vars['sessionurl'] . "do=removedupeposts&startat=$finishat&pp=" . $vbulletin->GPC['perpage'] . "\">" . $vbphrase['click_here_to_continue_processing'] . "</a></p>";
}
else
{
define('CP_REDIRECT', 'misc.php');
print_stop_message('deleted_duplicate_threads_successfully');
}
}
// /Delete Duplicate Posts Hack by vBulletinUsers.com
One final important note: This is only of benefit to people who, for one reason or another, may end up with duplicate posts in their vBulletin tables. The NNTP hack is one example of it might occur. If you do not need this functionality then I would recommend that you do not install this hack then! There is absolutely no point in adding a hack to your vBulletin community that has the potential of removing posts from your threads unless you are absolutely 100% positive that you need it.