The branch, master has been updated via 428eb4599f1aad8c4c5b71d02b91252a1daba24d (commit) via 089ed434abbcbbf03e7732ad362162a55c0e396e (commit) via 94c99b69df4bb1d9b4859b4796c99d0a4d9d4ae4 (commit) via ded1b3d0daf464ae240e9e088edc30986992537a (commit) via 7b81a1095047ef5126801584fae1e6b9312f9558 (commit) via 440928d5e9210df53eb54101a4d876d99c2ad531 (commit) via 876d0c2a281d0198ed392a5ba5a307e1b200b132 (commit) via aa087e76db76df3c8e8a9e052aefcb99371880f6 (commit) via 5257767d84ebdc3c2c766bcecee6c046f5e78d5d (commit) via cebcfe4a025c588c5f9197b706a1b3c0ca61653e (commit) via 337e79cfd28d94749e681048fb8b32441ba37ff5 (commit) via 45c0f862895dfe537b2ed6e55cca35b4565abe63 (commit) via c1e75d95c13f6eee442bbe0e383aa3944a2cbe3b (commit) via dda4ac413007e12639f709e8e1a41c3583e78e36 (commit) via de4869db73405466eec3f35456231ee051b1cb3d (commit) via 32c86fe1d577f9710bdc1edb34714780ea145d0c (commit) via 69c0b42f8328d3fb131d7658872313d377a5acc3 (commit) via ede5f340a093569596670be592d4aace355df05a (commit) via 3b6ca55f3178b27eec3e96c430e541aa9e080791 (commit) via b3380601fced230e566b5fee2f21db826f999f51 (commit) via e2215900b43a05781a2ab0aa1394863f851c4701 (commit) via 70959f30862c73643bbe2a6b62711ff742cc29f7 (commit) via d1cb3d4f7526a5103cffd722957bb7f8b323519e (commit) via 46bcc11ec89bfe72d7fa11fff0cb762b346b8a03 (commit) via e21d955ffd42e6f59c67cd52681b782d40cf41a9 (commit) via dd641f972d763dd79422ea216f4024dc3679e34c (commit) via 09cf3ac4a9ef1c2fad6b3b5585ae65316960e837 (commit) from d1fb15bad755ce96c840a5b1fa37986353d41e1a (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 428eb4599f1aad8c4c5b71d02b91252a1daba24d Author: Don Armstrong <don@donarmstrong.com> Date: Mon May 6 16:01:03 2013 -0700 accept -1 in addition to 0 for the current message in summary/outlook; properly handle control psuedoheaders commit 089ed434abbcbbf03e7732ad362162a55c0e396e Author: Don Armstrong <don@donarmstrong.com> Date: Fri Apr 19 13:27:10 2013 -0700 encode all strings to bytes before including in estraier commit 94c99b69df4bb1d9b4859b4796c99d0a4d9d4ae4 Author: Don Armstrong <don@donarmstrong.com> Date: Thu Apr 4 10:25:02 2013 -0700 Text::Iconv has a bug where it fails to reset the converter with a call to iconv(cd,NULL,(size_t) 0,NULL,(size_t) 0) if it fails during conversion; always create a new Text::Iconv object to work around this commit ded1b3d0daf464ae240e9e088edc30986992537a Author: Don Armstrong <don@donarmstrong.com> Date: Tue Apr 2 10:13:22 2013 -0700 check the log modified time of an optional gziped log commit 7b81a1095047ef5126801584fae1e6b9312f9558 Author: Don Armstrong <don@donarmstrong.com> Date: Fri Mar 22 12:13:41 2013 -0700 use a div instead of p for the headers; use headerfield for the field areas; increase padding for pre.message commit 440928d5e9210df53eb54101a4d876d99c2ad531 Merge: 876d0c2 aa087e7 Author: Don Armstrong <don@donarmstrong.com> Date: Thu Mar 21 15:08:54 2013 -0700 Merge branch 'don/rebuild-index' commit 876d0c2a281d0198ed392a5ba5a307e1b200b132 Author: Don Armstrong <don@donarmstrong.com> Date: Thu Mar 21 15:08:40 2013 -0700 fix pod test to not look at editor temp files commit aa087e76db76df3c8e8a9e052aefcb99371880f6 Author: Don Armstrong <don@donarmstrong.com> Date: Thu Mar 21 15:02:41 2013 -0700 change convert_to_utf8 to attempt to fix doubly encoded things commit 5257767d84ebdc3c2c766bcecee6c046f5e78d5d Author: Don Armstrong <don@donarmstrong.com> Date: Thu Mar 21 14:41:33 2013 -0700 fix unterminated B<> sequence sort bugs by number in rebuild it's index.db.realtime, not index.db-h.realtime. commit cebcfe4a025c588c5f9197b706a1b3c0ca61653e Merge: 45c0f86 337e79c Author: Don Armstrong <don@donarmstrong.com> Date: Thu Mar 21 10:39:04 2013 -0700 Merge branch 'don/rebuild-index' commit 337e79cfd28d94749e681048fb8b32441ba37ff5 Author: Don Armstrong <don@donarmstrong.com> Date: Wed Mar 20 17:54:50 2013 -0700 make generate_status_line handle bug not being defined in data commit 45c0f862895dfe537b2ed6e55cca35b4565abe63 Merge: c1e75d9 de4869d Author: Don Armstrong <don@donarmstrong.com> Date: Wed Mar 20 13:41:00 2013 -0700 Merge remote-tracking branch 'mikegabriel/proposal_upstream' into mike_patches commit c1e75d95c13f6eee442bbe0e383aa3944a2cbe3b Author: Don Armstrong <don@donarmstrong.com> Date: Wed Mar 20 13:37:20 2013 -0700 when we fail to commit to utf8, output the charset and data commit dda4ac413007e12639f709e8e1a41c3583e78e36 Author: Don Armstrong <don@donarmstrong.com> Date: Wed Mar 20 07:31:12 2013 -0700 add rebuild index support commit de4869db73405466eec3f35456231ee051b1cb3d Author: Mike Gabriel <mike.gabriel@das-netzwerkteam.de> Date: Wed Mar 20 12:07:42 2013 +0100 /debian/control: de-capitalize description synopsis lines commit 32c86fe1d577f9710bdc1edb34714780ea145d0c Author: Mike Gabriel <mike.gabriel@das-netzwerkteam.de> Date: Tue Mar 19 11:48:56 2013 +0100 Pass param bug=<NNN> to bugreport.html call in bugreport.cgi. commit 69c0b42f8328d3fb131d7658872313d377a5acc3 Author: Mike Gabriel <mike.gabriel@das-netzwerkteam.de> Date: Mon Mar 18 11:48:44 2013 +0100 Raise Standards to 3.9.4 (after fixing several lintian issues). commit ede5f340a093569596670be592d4aace355df05a Author: Mike Gabriel <mike.gabriel@das-netzwerkteam.de> Date: Mon Mar 18 11:36:27 2013 +0100 * Fix lintian issues: libdebbugs-perl: spelling-error-in-manpage for: + usr/share/man/man3/Debbugs::Recipients.3pm.gz + usr/share/man/man3/Debbugs::SOAP.3pm.gz + usr/share/man/man3/Debbugs::Status.3pm.gz commit 3b6ca55f3178b27eec3e96c430e541aa9e080791 Author: Mike Gabriel <mike.gabriel@das-netzwerkteam.de> Date: Mon Mar 18 11:42:48 2013 +0100 Fix lintian issues: debbugs source: debian-rules-makemaker-prefix-is-deprecated line 44, debian-rules-makemaker-prefix-is-deprecated line 56. commit b3380601fced230e566b5fee2f21db826f999f51 Author: Mike Gabriel <mike.gabriel@das-netzwerkteam.de> Date: Mon Mar 18 11:27:57 2013 +0100 Fix lintian issue: debbugs source: duplicate-in-relation-field in libdebbugs-perl depends: libtext-template-perl, libtext-template-perl. commit e2215900b43a05781a2ab0aa1394863f851c4701 Author: Mike Gabriel <mike.gabriel@das-netzwerkteam.de> Date: Mon Mar 18 11:14:01 2013 +0100 Switch to debian/source/format: 3.0 (native). commit 70959f30862c73643bbe2a6b62711ff742cc29f7 Author: Mike Gabriel <mike.gabriel@das-netzwerkteam.de> Date: Mon Mar 18 11:13:01 2013 +0100 Fix lintian issues: debian-rules-missing-recommended-target build-arch, debian-rules-missing-recommended-target build-indep. commit d1cb3d4f7526a5103cffd722957bb7f8b323519e Author: Mike Gabriel <mike.gabriel@das-netzwerkteam.de> Date: Mon Mar 18 11:10:48 2013 +0100 Fix lintian issue: description-synopsis-starts-with-article, also: capitalize first letter in all description synopsis lines. commit 46bcc11ec89bfe72d7fa11fff0cb762b346b8a03 Author: Mike Gabriel <mike.gabriel@das-netzwerkteam.de> Date: Mon Mar 18 11:08:52 2013 +0100 have a length < 0 (closes: #688344). * Fix spelling error in /debian/changelog. commit e21d955ffd42e6f59c67cd52681b782d40cf41a9 Author: Don Armstrong <don@donarmstrong.com> Date: Tue Mar 19 15:34:17 2013 -0700 add first work on rebuild index.db commit dd641f972d763dd79422ea216f4024dc3679e34c Author: Don Armstrong <don@donarmstrong.com> Date: Tue Mar 19 15:19:58 2013 -0700 write to bug closers in UTF-8 commit 09cf3ac4a9ef1c2fad6b3b5585ae65316960e837 Author: Don Armstrong <don@donarmstrong.com> Date: Mon Mar 18 10:42:15 2013 -0700 Fix double encoding of attachments (closes: #703300) ----------------------------------------------------------------------- Summary of changes: Debbugs/CGI/Bugreport.pm | 77 +++++++------- Debbugs/Control.pm | 6 +- Debbugs/Estraier.pm | 15 ++- Debbugs/Recipients.pm | 2 +- Debbugs/SOAP.pm | 2 +- Debbugs/Status.pm | 55 +++++++--- Debbugs/UTF8.pm | 47 +++++---- bin/debbugs-rebuild-index.db | 157 ++++++++++++++++++++++++++++ debian/changelog | 19 +++- debian/control | 7 +- debian/rules | 12 ++- debian/source/format | 1 + examples/debian/postpa/21bugclosers | 1 + html/bugs.css | 13 ++- t/01_pod.t | 4 +- t/{13_utf8_mail.t => 15_rebuild_indexdb.t} | 13 +-- templates/en_US/cgi/bugreport.tmpl | 2 +- 17 files changed, 324 insertions(+), 109 deletions(-) create mode 100755 bin/debbugs-rebuild-index.db create mode 100644 debian/source/format copy t/{13_utf8_mail.t => 15_rebuild_indexdb.t} (83%) The diff of changes is: diff --git a/Debbugs/CGI/Bugreport.pm b/Debbugs/CGI/Bugreport.pm index b3f4ec6..d9a8744 100644 --- a/Debbugs/CGI/Bugreport.pm +++ b/Debbugs/CGI/Bugreport.pm @@ -148,10 +148,10 @@ sub display_entity { if ($_ eq 'From' and $param{avatars}) { my $libravatar_url = __libravatar_url(decode_rfc1522($head_field)); if (defined $libravatar_url and length $libravatar_url) { - push @headers,q(<img src=").$libravatar_url.qq(">\n); + push @headers,q(<img src=").$libravatar_url.qq(" alt="">\n); } } - push @headers, qq(<p><span class="header">$_:</span> ) . html_escape(decode_rfc1522($head_field))."</p>\n"; + push @headers, qq(<div class="header"><span class="headerfield">$_:</span> ) . html_escape(decode_rfc1522($head_field))."</div>\n"; } print {$output} join(qq(), @headers); } else { @@ -172,7 +172,6 @@ sub display_entity { my $body = $entity->stringify_body; # this attachment has its own content type, so we must not # try to convert it to UTF-8 or do anything funky. - my @layers = PerlIO::get_layers($output); binmode($output,':raw'); print {$output} "Content-Type: $type"; my ($charset) = $head->get('Content-Type:') =~ m/charset\s*=\s*\"?([\w-]+)\"?/i; @@ -187,10 +186,10 @@ sub display_entity { print {$output} "\n"; my $decoder = MIME::Decoder->new($head->mime_encoding); $decoder->decode(IO::Scalar->new(\$body), $output); - if (grep {/utf8/} @layers) { - binmode($output,':utf8'); - } - return; + # we don't reset the layers here, because it makes no + # sense to add anything to the output handle after this + # point. + return(1); } elsif (not exists $param{att}) { my @dlargs = (msg=>$xmessage, att=>$#$attachments); @@ -206,26 +205,30 @@ sub display_entity { } } - return if not $param{outer} and $disposition eq 'attachment' and not exists $param{att}; - return unless ($type =~ m[^text/?] and - $type !~ m[^text/(?:html|enriched)(?:;|$)]) or - $type =~ m[^application/pgp(?:;|$)] or - $entity->parts; + return 0 if not $param{outer} and $disposition eq 'attachment' and not exists $param{att}; + return 0 unless (($type =~ m[^text/?] and + $type !~ m[^text/(?:html|enriched)(?:;|$)]) or + $type =~ m[^application/pgp(?:;|$)] or + $entity->parts); if ($entity->is_multipart) { my @parts = $entity->parts; foreach my $part (@parts) { - display_entity(entity => $part, - bug_num => $ref, - outer => 0, - msg_num => $xmessage, - output => $output, - attachments => $attachments, - terse => $param{terse}, - exists $param{msg}?(msg=>$param{msg}):(), - exists $param{att}?(att=>$param{att}):(), - exists $param{avatars}?(avatars=>$param{avatars}):(), - ); + my $raw_output = + display_entity(entity => $part, + bug_num => $ref, + outer => 0, + msg_num => $xmessage, + output => $output, + attachments => $attachments, + terse => $param{terse}, + exists $param{msg}?(msg=>$param{msg}):(), + exists $param{att}?(att=>$param{att}):(), + exists $param{avatars}?(avatars=>$param{avatars}):(), + ); + if ($raw_output) { + return $raw_output; + } # print {$output} "\n"; } } elsif ($entity->parts) { @@ -286,6 +289,7 @@ sub display_entity { print {$output} qq(<pre class="message">$body</pre>\n); } } + return 0; } @@ -314,19 +318,20 @@ sub handle_email_message{ $parser->output_to_core(1); my $entity = $parser->parse_data( $email); my @attachments = (); - display_entity(entity => $entity, - bug_num => $param{ref}, - outer => 1, - msg_num => $param{msg_num}, - output => $output_fh, - attachments => \@attachments, - terse => $param{terse}, - exists $param{msg}?(msg=>$param{msg}):(), - exists $param{att}?(att=>$param{att}):(), - exists $param{trim_headers}?(trim_headers=>$param{trim_headers}):(), - exists $param{avatars}?(avatars=>$param{avatars}):(), - ); - return decode_utf8($output); + my $raw_output = + display_entity(entity => $entity, + bug_num => $param{ref}, + outer => 1, + msg_num => $param{msg_num}, + output => $output_fh, + attachments => \@attachments, + terse => $param{terse}, + exists $param{msg}?(msg=>$param{msg}):(), + exists $param{att}?(att=>$param{att}):(), + exists $param{trim_headers}?(trim_headers=>$param{trim_headers}):(), + exists $param{avatars}?(avatars=>$param{avatars}):(), + ); + return $raw_output?$output:decode_utf8($output); } =head2 handle_record diff --git a/Debbugs/Control.pm b/Debbugs/Control.pm index 0390ba7..44d0062 100644 --- a/Debbugs/Control.pm +++ b/Debbugs/Control.pm @@ -2663,7 +2663,7 @@ Handles all setting of summary fields If summary is undef, unsets the summary -If summary is 0, sets the summary to the first paragraph contained in +If summary is 0 or -1, sets the summary to the first paragraph contained in the message passed. If summary is a positive integer, sets the summary to the message specified. @@ -2751,7 +2751,7 @@ sub _summary { elsif ($param{$cmd} =~ /^\d+$/) { my $log = []; my @records = Debbugs::Log::read_log_records(bug_num => $param{bug}); - if ($param{$cmd} == 0) { + if ($param{$cmd} == 0 or $param{$cmd} == -1) { $log = $param{message}; $summary_msg = @records + 1; } @@ -2785,7 +2785,7 @@ sub _summary { } # skip a paragraph if it looks like it's control or # pseudo-headers - if ($line =~ m{^\s*(?:Package|Source|Version|User|Tag|Severity)\:\s+\S}xi or #pseudo headers + if ($line =~ m{^\s*(?:Package|Source|Version|User|Tag|Severity|Control)\:\s+\S}xi or #pseudo headers $line =~ m{^(?:package:?|(?:no|)owner|severity|tags?|summary| #control \#|reopen|close|(?:not|)(?:fixed|found)|clone| debug|(?:not|)forwarded|priority| diff --git a/Debbugs/Estraier.pm b/Debbugs/Estraier.pm index e3c89a1..5987fe9 100644 --- a/Debbugs/Estraier.pm +++ b/Debbugs/Estraier.pm @@ -30,12 +30,11 @@ use strict; use vars qw($VERSION $DEBUG %EXPORT_TAGS @EXPORT_OK @EXPORT); use base qw(Exporter); use Debbugs::Log; -#use Params::Validate; use Search::Estraier; -use Date::Manip; use Debbugs::Common qw(getbuglocation getbugcomponent make_list); use Debbugs::Status qw(readbug); use Debbugs::MIME qw(parse); +use Encode qw(encode_utf8); BEGIN{ ($VERSION) = q$Revision: 1.3 $ =~ /^Revision:\s+([^\s+])/; @@ -121,7 +120,7 @@ sub add_bug_message{ $doc = new Search::Estraier::Document if not defined $doc; my $message = parse($bug_message); - $doc->add_text(join("\n",make_list(values %{$message}))); + $doc->add_text(encode_utf8(join("\n",make_list(values %{$message})))); # * @id : the ID number determined automatically when the document is registered. # * @uri : the location of a document which any document should have. @@ -140,17 +139,17 @@ sub add_bug_message{ my @attr = qw(status subject date submitter package tags severity); # parse the date my ($date) = $bug_message =~ /^Date:\s+(.+?)\s*$/mi; - $doc->add_attr('@cdate' => $date) if defined $date; + $doc->add_attr('@cdate' => encode_utf8($date)) if defined $date; # parse the title my ($subject) = $bug_message =~ /^Subject:\s+(.+?)\s*$/mi; - $doc->add_attr('@title' => $subject) if defined $subject; + $doc->add_attr('@title' => encode_utf8($subject)) if defined $subject; # parse the author my ($author) = $bug_message =~ /^From:\s+(.+?)\s*$/mi; - $doc->add_attr('@author' => $author) if defined $author; + $doc->add_attr('@author' => encode_utf8($author)) if defined $author; # create the uri - $doc->add_attr('@uri' => $uri); + $doc->add_attr('@uri' => encode_utf8($uri)); foreach my $attr (@attr) { - $doc->add_attr($attr => $status->{$attr}) if defined $status->{$attr}; + $doc->add_attr($attr => encode_utf8($status->{$attr})) if defined $status->{$attr}; } print STDERR "adding $uri\n" if $DEBUG; # Try a bit harder if estraier is returning timeouts diff --git a/Debbugs/Recipients.pm b/Debbugs/Recipients.pm index 47c6485..ae739e9 100644 --- a/Debbugs/Recipients.pm +++ b/Debbugs/Recipients.pm @@ -245,7 +245,7 @@ Using the recipient hashref, determines the set of recipients. If you specify one of C<bcc>, C<cc>, or C<to>, you will receive only a LIST of recipients which the main should be Bcc'ed, Cc'ed, or To'ed respectively. By default, a LIST with keys bcc, cc, and to is returned -with ARRAYREF values correponding to the users to whom a message +with ARRAYREF values corresponding to the users to whom a message should be sent. =over diff --git a/Debbugs/SOAP.pm b/Debbugs/SOAP.pm index c1fc85f..f87581e 100644 --- a/Debbugs/SOAP.pm +++ b/Debbugs/SOAP.pm @@ -343,7 +343,7 @@ architectures are at which versions. =back -This function correponds to L<Debbugs::Packages::get_versions> +This function corresponds to L<Debbugs::Packages::get_versions> =cut diff --git a/Debbugs/Status.pm b/Debbugs/Status.pm index 9c9551b..bf3d2ab 100644 --- a/Debbugs/Status.pm +++ b/Debbugs/Status.pm @@ -70,6 +70,7 @@ BEGIN{ qw(removefoundversions removefixedversions) ], hook => [qw(bughook bughook_archive)], + indexdb => [qw(generate_index_db_line)], fields => [qw(%fields)], ); @EXPORT_OK = (); @@ -207,6 +208,7 @@ sub read_bug{ $log = $status; $log =~ s/\.summary$/.log/; ($location) = $status =~ m/(db-h|db|archive)/; + ($param{bug}) = $status =~ m/(\d+)\.summary$/; } if ($param{lock}) { filelock("$config{spool_dir}/lock/$param{bug}",exists $param{locks}?$param{locks}:()); @@ -274,7 +276,7 @@ sub read_bug{ my $status_modified = (stat($status))[9]; # Add log last modified time - $data{log_modified} = (stat($log))[9]; + $data{log_modified} = (stat($log))[9] // (stat("${log}.gz"))[9]; $data{last_modified} = max($status_modified,$data{log_modified}); $data{location} = $location; $data{archived} = (defined($location) and ($location eq 'archive'))?1:0; @@ -655,7 +657,7 @@ sub makestatus { Writes the bug status and summary files out. -Skips writting out a status file if minversion is 2 +Skips writing out a status file if minversion is 2 Does not call bughook if disablebughook is true. @@ -773,7 +775,7 @@ exactly are removed. Otherwise, all versions matching the version number are removed. Currently $package and $isbinary are entirely ignored, but accepted -for backwards compatibilty. +for backwards compatibility. =cut @@ -1599,6 +1601,39 @@ sub isstrongseverity { return grep { $_ eq $severity } @{$config{strong_severities}}; } +=head1 indexdb + +=head2 generate_index_db_line + + my $data = read_bug(bug => $bug, + location => $initialdir); + # generate_index_db_line hasn't been written yet at all. + my $line = generate_index_db_line($data); + +Returns a line for a bug suitable to be written out to index.db. + +=cut + +sub generate_index_db_line { + my ($data,$bug) = @_; + + # just in case someone has given us a split out data + $data = join_status_fields($data); + + my $whendone = "open"; + my $severity = $config{default_severity}; + (my $pkglist = $data->{package}) =~ s/[,\s]+/,/g; + $pkglist =~ s/^,+//; + $pkglist =~ s/,+$//; + $whendone = "forwarded" if defined $data->{forwarded} and length $data->{forwarded}; + $whendone = "done" if defined $data->{done} and length $data->{done}; + $severity = $data->{severity} if length $data->{severity}; + return sprintf "%s %d %d %s [%s] %s %s\n", + $pkglist, $data->{bug_num}//$bug, $data->{date}, $whendone, + $data->{originator}, $severity, $data->{keywords}; +} + + =head1 PRIVATE FUNCTIONS @@ -1681,19 +1716,7 @@ sub bughook { my $data = $bugs_temp{$bug}; appendfile("$config{spool_dir}/debbugs.trace","$type $bug\n",makestatus($data, 1)); - my $whendone = "open"; - my $severity = $config{default_severity}; - (my $pkglist = $data->{package}) =~ s/[,\s]+/,/g; - $pkglist =~ s/^,+//; - $pkglist =~ s/,+$//; - $whendone = "forwarded" if defined $data->{forwarded} and length $data->{forwarded}; - $whendone = "done" if defined $data->{done} and length $data->{done}; - $severity = $data->{severity} if length $data->{severity}; - - my $k = sprintf "%s %d %d %s [%s] %s %s\n", - $pkglist, $bug, $data->{date}, $whendone, - $data->{originator}, $severity, $data->{keywords}; - $bugs{$bug} = $k; + $bugs{$bug} = generate_index_db_line($data,$bug); } update_realtime("$config{spool_dir}/index.db.realtime", %bugs); diff --git a/Debbugs/UTF8.pm b/Debbugs/UTF8.pm index 45da58f..bc3a0fa 100644 --- a/Debbugs/UTF8.pm +++ b/Debbugs/UTF8.pm @@ -143,10 +143,9 @@ sub decode_utf8_safely{ =cut -our %iconv_converters; - sub convert_to_utf8 { - my ($data,$charset) = @_; + my ($data,$charset,$internal_call) = @_; + $internal_call //= 0; if (is_utf8($data)) { cluck("utf8 flag is set when calling convert_to_utf8"); return $data; @@ -155,30 +154,36 @@ sub convert_to_utf8 { if ($charset eq 'RAW') { croak("Charset must not be raw when calling convert_to_utf8"); } - if (not defined $iconv_converters{$charset}) { - eval { - $iconv_converters{$charset} = Text::Iconv->new($charset,"UTF-8") or - die "Unable to create converter for '$charset'"; - }; - if ($@) { - warn $@; - # We weren't able to create the converter, so use Encode - # instead - return __fallback_convert_to_utf8($data,$charset); - } - } - if (not defined $iconv_converters{$charset}) { - warn "The converter for $charset wasn't created properly somehow!"; + my $iconv_converter; + eval { + $iconv_converter = Text::Iconv->new($charset,"UTF-8") or + die "Unable to create converter for '$charset'"; + }; + if ($@) { + return undef if $internal_call; + warn $@; + # We weren't able to create the converter, so use Encode + # instead return __fallback_convert_to_utf8($data,$charset); } - my $converted_data = $iconv_converters{$charset}->convert($data); + my $converted_data = $iconv_converter->convert($data); # if the conversion failed, retval will be undefined or perhaps # -1. - my $retval = $iconv_converters{$charset}->retval(); + my $retval = $iconv_converter->retval(); if (not defined $retval or $retval < 0 ) { - warn "failed to convert to utf8"; + # try iso8559-1 first + if (not $internal_call) { + my $call_back_data = convert_to_utf8($data,'ISO8859-1',1); + # if there's an à (0xC3), it's probably something + # horrible, and we shouldn't try to convert it. + if (defined $call_back_data and $call_back_data !~ /\x{C3}/) { + warn "failed to convert to utf8 (charset: $charset, data: $data), but succeeded with ISO8859-1: ".encode_utf8($call_back_data); + return $call_back_data; + } + } + warn "failed to convert to utf8 (charset: $charset, data: $data)"; # Fallback to encode, which will probably also fail. return __fallback_convert_to_utf8($data,$charset); } @@ -199,7 +204,7 @@ sub __fallback_convert_to_utf8 { $charset //= 'utf8'; my $result; eval { - $result = decode($charset,$data); + $result = decode($charset,$data,0); }; if ($@) { warn "Unable to decode charset; '$charset' and '$data': $@"; diff --git a/bin/debbugs-rebuild-index.db b/bin/debbugs-rebuild-index.db new file mode 100755 index 0000000..c44bc32 --- /dev/null +++ b/bin/debbugs-rebuild-index.db @@ -0,0 +1,157 @@ +#! /usr/bin/perl +# debbugs-rebuild-index.db is part of debbugs, and is released +# under the terms of the GPL version 2, or any later version, at your +# option. See the file README and COPYING for more information. +# Copyright 2012 by Don Armstrong <don@donarmstrong.com>. + + +use warnings; +use strict; + +use Getopt::Long qw(:config no_ignore_case); +use Pod::Usage; + +=head1 NAME + +debbugs-rebuild-index.db -- rebuild Debbug's index.db + +=head1 SYNOPSIS + +debbugs-rebuild-index.db [options] + + Options: + --spool-dir debbugs spool directory + --debug, -d debugging level (Default 0) + --help, -h display this help + --man, -m display manual + +=head1 OPTIONS + +=over + +=item B<--spool-dir> + +Debbugs spool directory; defaults to the value configured in the +debbugs configuration file. + +=item B<--debug, -d> + +Debug verbosity. + +=item B<--help, -h> + +Display brief useage information. + +=item B<--man, -m> + +Display this manual. + +=back + +=head1 EXAMPLES + +Rebuild the index.db for db-h. + + debbugs-rebuild-index.db; + +Rebuild the index.db for archive + + debbugs-rebuild-index.db archive; + + +=cut + + +use vars qw($DEBUG); + +use Debbugs::Common qw(checkpid lockpid get_hashname getparsedaddrs getbugcomponent make_list); +use Debbugs::Config qw(:config); +use Debbugs::Status qw(read_bug split_status_fields generate_index_db_line); + +my %options = (debug => 0, + help => 0, + man => 0, + verbose => 0, + quiet => 0, + quick => 0, + service => 'debbugs', + ); + + +GetOptions(\%options, + 'quick|q', + 'service|s', + 'sysconfdir|c', + 'spool_dir|spool-dir=s', + 'debug|d+','help|h|?','man|m'); + +pod2usage() if $options{help}; +pod2usage({verbose=>2}) if $options{man}; + +$DEBUG = $options{debug}; + +my @USAGE_ERRORS; +$options{verbose} = $options{verbose} - $options{quiet}; + +pod2usage(join("\n",@USAGE_ERRORS)) if @USAGE_ERRORS; + +if (exists $options{spool_dir} and defined $options{spool_dir}) { + $config{spool_dir} = $options{spool_dir}; +} +chdir($config{spool_dir}) or die "chdir $config{spool_dir} failed: $!"; + +my $verbose = $options{debug}; + +my $initialdir = "db-h"; + +if (defined $ARGV[0] and $ARGV[0] eq "archive") { + $initialdir = "archive"; +} + +if (not lockpid($config{spool_dir}.'/lock/debbugs-rebuild-index.db')) { + print STDERR "Another debbugs-rebuild-index.db is running; stopping\n"; + exit 1; +} + +my $fh_type = $initialdir; +# if initaldir is db-h, the file is db. +$fh_type = 'db' if $initialdir eq 'db-h'; + +my $file = "index.${fh_type}.realtime"; +my $idx_rebuild = IO::File->new($file.'.rebuild','w') + or die "Couldn't open ${file}.rebuild: $!"; + + +my @dirs = ($initialdir); +my $cnt = 0; +my %bugs; +while (my $dir = shift @dirs) { + printf "Doing dir %s ...\n", $dir if $verbose; + + opendir(DIR, "$dir/.") or die "opendir $dir: $!"; + my @subdirs = readdir(DIR); + closedir(DIR); + + my @list = map { m/^(\d+)\.summary$/?($1):() } @subdirs; + push @dirs, map { m/^(\d+)$/ && -d "$dir/$1"?("$dir/$1"):() } @subdirs; + + for my $bug (@list) { + print "Up to $cnt bugs...\n" if (++$cnt % 100 == 0 && $verbose); + my $stat = stat(getbugcomponent($bug,'summary',$initialdir)); + if (not defined $stat) { + print STDERR "Unable to stat $bug $!\n"; + next; + } + my $data = read_bug(bug => $bug, + location => $initialdir); + my $line = generate_index_db_line($data); + $bugs{$bug} = $line; + } +} +binmode($idx_rebuild,':raw:encoding(UTF-8)'); +print {$idx_rebuild} $bugs{$_} foreach sort {$a <=> $b} keys %bugs; +close($idx_rebuild); +rename("$file.rebuild", $file); + + +__END__ diff --git a/debian/changelog b/debian/changelog index 14a0d66..e2ab47d 100644 --- a/debian/changelog +++ b/debian/changelog @@ -17,6 +17,7 @@ debbugs (2.4.2~exp2) UNRELEASED; urgency=low * Fix failure to forcibly merge/merge when found/fixed is not qualified (closes: #670456). Thanks to Jonathan Nieder and Bernhard Schmidt. * Add libravatar support. + * Fix double encoding of attachments (closes: #703300) [Thanks to Arnout Engelen: ] * Add Homepage (closes: #670555). @@ -52,9 +53,25 @@ debbugs (2.4.2~exp2) UNRELEASED; urgency=low a site admin aware of its existence (closes: #688345). * Make sure that mails to gSubscriptionDomain and gBugSubscriptionDomain are only sent out if the variables in config are defined and - have a lenght < 0 (closes: #688344). + have a length < 0 (closes: #688344). * Use locale independent date format for mail processing and service mail generation (closes: #688745). + * Fix spelling error in /debian/changelog. + * Fix lintian issue: description-synopsis-starts-with-article. + * Fix lintian issues: debian-rules-missing-recommended-target build-arch, + debian-rules-missing-recommended-target build-indep. + * Switch to debian/source/format: 3.0 (native). + * Fix lintian issue: debbugs source: duplicate-in-relation-field in + libdebbugs-perl depends: libtext-template-perl, libtext-template-perl. + * Fix lintian issues: debbugs source: + debian-rules-makemaker-prefix-is-deprecated line 44, + debian-rules-makemaker-prefix-is-deprecated line 56. + * Fix lintian issues: libdebbugs-perl: spelling-error-in-manpage for: + + usr/share/man/man3/Debbugs::Recipients.3pm.gz + + usr/share/man/man3/Debbugs::SOAP.3pm.gz + + usr/share/man/man3/Debbugs::Status.3pm.gz + * Raise Standards to 3.9.4 (after fixing several lintian issues). + * Pass param bug=<NNN> to bugreport.html call in bugreport.cgi. -- Don Armstrong <don@debian.org> Wed, 25 Aug 2010 01:57:38 -0700 diff --git a/debian/control b/debian/control index f0f5d61..d8f19b5 100644 --- a/debian/control +++ b/debian/control @@ -3,7 +3,7 @@ Section: misc Priority: extra Maintainer: Debbugs developers <debian-debbugs@lists.debian.org> Uploaders: Colin Watson <cjwatson@debian.org>, Don Armstrong <don@debian.org> -Standards-Version: 3.8.2 +Standards-Version: 3.9.4 Vcs-Browser: http://bugs.debian.org/debbugs-source/mainline Vcs-Git: http://bugs.debian.org/debbugs-source/debbugs.git Build-Depends: debhelper (>= 5) @@ -25,7 +25,7 @@ Depends: libdebbugs-perl, Recommends: debbugs-web Suggests: spamassassin (>= 3.0), libcgi-alert-perl -Description: The bug tracking system based on the active Debian BTS +Description: bug tracking system based on the active Debian BTS Debian has a bug tracking system which files details of bugs reported by users and developers. Each bug is given a number, and is kept on file until it is marked as having been dealt with. The system is mainly controlled by @@ -44,7 +44,6 @@ Depends: libio-stringy-perl, libmldbm-perl, liburi-perl, libsoap-lite-perl, libcgi-simple-perl, libparams-validate-perl, libtext-template-perl, libsafe-hole-perl, libmail-rfc822-address-perl, liblist-moreutils-perl, - libtext-template-perl Section: perl Description: modules used by the active Debian BTS Debian has a bug tracking system which files details of bugs reported by @@ -77,7 +76,7 @@ Depends: ${misc:Depends}, libdebbugs-perl, debbugs-web, libconfig-simple-perl, libuser-perl, rsync, libhttp-server-simple-perl, libnet-server-perl -Description: run and maintains a local mirror of the Debian BTS +Description: Run and maintains a local mirror of the Debian BTS Debian has a bug tracking system which files details of bugs reported by users and developers. Each bug is given a number, and is kept on file until it is marked as having been dealt with. The system is diff --git a/debian/rules b/debian/rules index c348379..7bdd341 100755 --- a/debian/rules +++ b/debian/rules @@ -10,7 +10,13 @@ DEST_DIR := $(CURDIR)/debian/tmp PERL ?= /usr/bin/perl -build: build-stamp +build: build-arch build-indep + +build-arch: +# nothing to do, as there aren't any architecture-dependent packages + +build-indep: build-stamp + build-stamp: # Call the test suite $(PERL) Makefile.PL INSTALLDIRS=vendor @@ -35,7 +41,7 @@ install-stamp: build dh_clean -k dh_installdirs $(MAKE) install_mostfiles DESTDIR=$(DEST_DIR) - $(MAKE) -f Makefile.perl install PREFIX=$(DEST_DIR)/usr + $(MAKE) -f Makefile.perl install DESTDIR=$(DEST_DIR) touch $@ binary-arch: @@ -47,7 +53,7 @@ binary-indep: build install dh_clean -k dh_installdirs $(MAKE) install_mostfiles DESTDIR=$(DEST_DIR) - $(MAKE) -f Makefile.perl install PREFIX=$(DEST_DIR)/usr + $(MAKE) -f Makefile.perl install DESTDIR=$(DEST_DIR) dh_install --sourcedir=debian/tmp --fail-missing dh_installdocs dh_installchangelogs diff --git a/debian/source/format b/debian/source/format new file mode 100644 index 0000000..9f67427 --- /dev/null +++ b/debian/source/format @@ -0,0 +1 @@ +3.0 (native) \ No newline at end of file diff --git a/examples/debian/postpa/21bugclosers b/examples/debian/postpa/21bugclosers index 382b779..dfcc44a 100755 --- a/examples/debian/postpa/21bugclosers +++ b/examples/debian/postpa/21bugclosers @@ -39,6 +39,7 @@ my %bugclosers = count_bugs(function => sub { open BUGCLOSERS, '> /org/bugs.debian.org/www/stats/bugclosers.txt.new' or die "can't open bugclosers.txt.new: $!"; +binmode(BUGCLOSERS,':raw:encoding(UTF-8)'); for my $closer (sort { $bugclosers{$a} <=> $bugclosers{$b} } keys %bugclosers) { printf BUGCLOSERS "%4d %s\n", $bugclosers{$closer}, $closer or die "can't write to bugclosers.txt.new: $!"; diff --git a/html/bugs.css b/html/bugs.css index c251cea..9019b9d 100644 --- a/html/bugs.css +++ b/html/bugs.css @@ -99,7 +99,7 @@ a.submitter:hover, a.submitter:visited:hover { pre.message { font-family: monospace; - padding-top: 0; + padding-top: 8px; margin-top: 0; border-top: 0; } @@ -133,6 +133,15 @@ div.headers { overflow: auto; } +div.header { + font-family: sans-serif; + font-size: 95%; + color: #3c3c3c; + padding: 0px; + line-height: 120%; + margin: 0px; +} + div.headers p { font-family: sans-serif; font-size: 95%; @@ -146,7 +155,7 @@ div.headers img { float: right; } -span.header { +span.headerfield { font-weight: bold }; diff --git a/t/01_pod.t b/t/01_pod.t index 8fb590f..2f1d2df 100644 --- a/t/01_pod.t +++ b/t/01_pod.t @@ -2,4 +2,6 @@ use Test::More; eval "use Test::Pod 1.00"; plan skip_all => "Test::Pod 1.00 required for testing POD" if $@; -all_pod_files_ok(all_pod_files('.')); +all_pod_files_ok(grep {$_ !~ /[~#]$/} all_pod_files((-e 'blib'?'blib':(qw(Debbugs Mail))), + (qw(bin cgi scripts)) + )); diff --git a/t/13_utf8_mail.t b/t/15_rebuild_indexdb.t similarity index 83% copy from t/13_utf8_mail.t copy to t/15_rebuild_indexdb.t index be6b6c6..a3036ec 100644 --- a/t/13_utf8_mail.t +++ b/t/15_rebuild_indexdb.t @@ -1,7 +1,6 @@ # -*- mode: cperl;-*- -# $Id: 05_mail.t,v 1.1 2005/08/17 21:46:17 don Exp $ -use Test::More tests => 13; +use Test::More tests => 9; use warnings; use strict; @@ -123,15 +122,7 @@ $SD_SIZE = $sendmail_dir, 'control@bugs.something messages appear to have been sent out properly'); # now we need to check to make sure the control message was processed without errors -ok(system('sh','-c','find '.$sendmail_dir.q( -type f | xargs grep -q "Subject: Processed: Munging a bug")) == 0, - 'control@bugs.something message was parsed without errors'); # now we need to check to make sure that the control message actually did anything # This is an eval because $ENV{DEBBUGS_CONFIG_FILE} isn't set at BEGIN{} time eval "use Debbugs::Status qw(read_bug writebug);"; -my $status = read_bug(bug=>1); -ok($status->{subject} eq 'ütff8 title encoding test','bug 1 retitled'); -ok($status->{severity} eq 'wishlist','bug 1 wishlisted'); -ok(system('sh','-c','[ $(egrep "retitle.*encoding test" '.$spool_dir.'/db-h/01/1.log|grep -v "=C3=BCt=EF=AC=808"|wc -l) -eq 0 ]') == 0, - 'Control messages escaped properly'); -ok(system('sh','-c',encode_utf8('grep -q "föoff@bugs.something" '.$spool_dir.'/index.db'))==0, - 'index.db not double escaped'); +ok(system('bin/debbugs-rebuild-index.db')==0,'debbugs-rebuild-index seems to work'); diff --git a/templates/en_US/cgi/bugreport.tmpl b/templates/en_US/cgi/bugreport.tmpl index 070005a..66a1ca2 100644 --- a/templates/en_US/cgi/bugreport.tmpl +++ b/templates/en_US/cgi/bugreport.tmpl @@ -43,7 +43,7 @@ function toggle_infmessages() } {$log} <hr> -<p class="msgreceived">Send a report that <a href="http://{$config{cgi_domain}}/bugspam.cgi">this bug log contains spam</a>.</p> +<p class="msgreceived">Send a report that <a href="http://{$config{cgi_domain}}/bugspam.cgi?bug={$bug_num}">this bug log contains spam</a>.</p> <hr> {include(q(html/html_tail))} </body> hooks/post-receive -- debbugs.git (debbugs upstream project) This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "debbugs.git" (debbugs upstream project).